Teradata Aster R User Guide Update 3

Transcription

1 What would you do if you knew? Teradata Aster R User Guide Update 3 Release B K December 2017

2 The product or products described in this book are licensed products of Teradata Corporation or its affiliates. Teradata, Aster, BYNET, Claraview, DecisionCast, IntelliBase, IntelliCloud, IntelliFlex, QueryGrid, SQL-MapReduce, Teradata Decision Experts, "Teradata Labs" logo, Teradata ServiceConnect, and Teradata Source Experts are trademarks or registered trademarks of Teradata Corporation or its affiliates in the United States and other countries. Adaptec and SCSISelect are trademarks or registered trademarks of Adaptec, Inc. Amazon Web Services, AWS, Amazon Elastic Compute Cloud, Amazon EC2, Amazon Simple Storage Service, Amazon S3, AWS CloudFormation, and AWS Marketplace are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries. AMD Opteron and Opteron are trademarks of Advanced Micro Devices, Inc. Apache, Apache Avro, Apache Hadoop, Apache Hive, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. Apple, Mac, and OS X all are registered trademarks of Apple Inc. Axeda is a registered trademark of Axeda Corporation. Axeda Agents, Axeda Applications, Axeda Policy Manager, Axeda Enterprise, Axeda Access, Axeda Software Management, Axeda Service, Axeda ServiceLink, and Firewall-Friendly are trademarks and Maximum Results and Maximum Support are servicemarks of Axeda Corporation. CENTOS is a trademark of Red Hat, Inc., registered in the U.S. and other countries. Cloudera and CDH are trademarks or registered trademarks of Cloudera Inc. in the United States, and in jurisdictions throughout the world. Data Domain, EMC, PowerPath, SRDF, and Symmetrix are either registered trademarks or trademarks of EMC Corporation in the United States and/or other countries. GoldenGate is a trademark of Oracle. Hewlett-Packard and HP are registered trademarks of Hewlett-Packard Company. Hortonworks, the Hortonworks logo and other Hortonworks trademarks are trademarks of Hortonworks Inc. in the United States and other countries. Intel, Pentium, and XEON are registered trademarks of Intel Corporation. IBM, CICS, RACF, Tivoli, IBM Spectrum Protect, and z/os are trademarks or registered trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. LSI is a registered trademark of LSI Corporation. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are registered trademarks of Microsoft Corporation in the United States and other countries. NetVault is a trademark of Quest Software, Inc. Novell and SUSE are registered trademarks of Novell, Inc., in the United States and other countries. Oracle, Java, and Solaris are registered trademarks of Oracle and/or its affiliates. QLogic and SANbox are trademarks or registered trademarks of QLogic Corporation. Quantum and the Quantum logo are trademarks of Quantum Corporation, registered in the U.S.A. and other countries. Red Hat is a trademark of Red Hat, Inc., registered in the U.S. and other countries. Used under license. SAP is the trademark or registered trademark of SAP AG in Germany and in several other countries. SAS and SAS/C are trademarks or registered trademarks of SAS Institute Inc. Sentinel is a registered trademark of SafeNet, Inc. Simba, the Simba logo, SimbaEngine, SimbaEngine C/S, SimbaExpress and SimbaLib are registered trademarks of Simba Technologies Inc. SPARC is a registered trademark of SPARC International, Inc. Unicode is a registered trademark of Unicode, Inc. in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Veritas, the Veritas Logo and NetBackup are trademarks or registered trademarks of Veritas Technologies LLC or its affiliates in the U.S. and other countries. Other product and company names mentioned herein may be the trademarks of their respective owners. The information contained in this document is provided on an "as-is" basis, without warranty of any kind, either express or implied, including the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. Some jurisdictions do not allow the exclusion of implied warranties, so the above exclusion may not apply to you. In no event will Teradata Corporation be liable for any indirect, direct, special, incidental, or consequential damages, including lost profits or lost savings, even if expressly advised of the possibility of such damages. The information contained in this document may contain references or cross-references to features, functions, products, or services that are not announced or available in your country. Such references do not imply that Teradata Corporation intends to announce such features, functions, products, or services in your country. Please consult your local Teradata Corporation representative for those features, functions, products, or services available in your country. Information contained in this document may contain technical inaccuracies or typographical errors. Information may be changed or updated without notice. Teradata Corporation may also make improvements or changes in the products or services described in this information at any time without notice. To maintain the quality of our products and services, we would like your comments on the accuracy, clarity, organization, and value of this document. Please teradata-books@lists.teradata.com Any comments or materials (collectively referred to as "Feedback") sent to Teradata Corporation will be deemed non-confidential. Teradata Corporation will have no obligation of any kind with respect to Feedback and will be free to use, reproduce, disclose, exhibit, display, transform, create derivative works of, and distribute the Feedback and derivative works thereof without limitation on a royalty-free basis. Further, Teradata Corporation will be free to use any ideas, concepts, know-how, or techniques contained in such Feedback for any purpose whatsoever, including developing, manufacturing, or marketing products or services incorporating Feedback. Copyright by Teradata. All Rights Reserved.

3 Preface Overview This guide provides instructions for users and administrators of version of the Teradata Aster R product. If a different Aster R version is used, users must download a different edition of this guide. The following additional resources are available: Aster Database upgrades, clients and other packages: Documentation for existing customers with a Your Service login: Documentation that is available to the public: Conventions Used in This Guide This document assumes that the reader is comfortable working in Windows and Linux/UNIX environments. Many sections assume the reader is familiar with SQL. This document uses the following typographical conventions. Typefaces Command line input and output, commands, program code, filenames, directory names, and system variables are shown in a monospaced font. Words in italics indicate an example or placeholder value that users must replace with a real value. Bold type is intended to draw users' attention to important or changed items. Menu navigation and user interface elements are shown using the User Interface Command font. Notation Conventions In the synopsis sections, these conventions are used: Square brackets ([ and ]) indicate one or more optional items. Curly braces ({ and }) indicate that users must choose an item from the list inside the braces. Choices are separated by vertical lines ( ). An ellipsis (...) means the preceding element can be repeated. A comma and an ellipsis (,...) means the preceding element can be repeated in a comma-separated list. In command line instructions, SQL commands and shell commands are typically written with no preceding prompt, but where needed the default SQL prompt is shown: beehive=> Teradata Aster R User Guide Update 3 3

4 Preface Contact Teradata Global Technical Support (GTS) Command Shell Text Conventions For shell commands, the prompt is usually shown. The $ sign introduces a command that is being run by a non-root user: $ ls The # sign introduces a command that is being run as root: # ls Contact Teradata Global Technical Support (GTS) For assistance and updated documentation, contact Teradata Global Technical Support (GTS): Support Portal: International: US Customers: 877-MyT-Data ( ) About Teradata Aster Teradata Aster provides data management and advanced analytics for diverse and big data, enabling the powerful combination of cost-effective storage and ultra-fast analysis of relational and non-relational data. Teradata Aster is a division of Teradata and is headquartered in Santa Clara, California. For more information, go to: About This Document This Teradata Aster R User Guide covers Aster R version , updated to support releases , and Version History Version History Table Release Product ID Date AA Updated to support AA release AA Updated to support AA release AA Updated to support AA release B K December 2017 B K September 2017 B K May 2017 AA B K May Teradata Aster R User Guide Update 3

5 Preface About This Document Release Product ID Date AC 6.22 Update B K October 2016 AC 6.22 Update B K August 2016 AC 6.22 B K June 2016 Teradata Aster R User Guide Update 3 5

6 Preface About This Document 6 Teradata Aster R User Guide Update 3

7 CHAPTER 1 Introduction to Aster R Introduction to Aster R The Teradata Aster R product combines the benefits of the open-source R environment with the massive parallel processing capabilities of the Teradata Aster R and analytic functions. Aster R allows users to develop and run R programs that take advantage of the big data analytics capabilities of the Aster Database. Aster R Product Overview The Aster R product includes two components: The TeradataAsterR package. This is a package like any other open-source R packages. The package provides an R interface for data exploration and analysis within the Aster Database. No data is transferred to the client machine. All of the Teradata Aster SQL-MapReduce functions are available through this interface with no SQL coding required. In addition, Aster R provides functions for data manipulation and transformation, data filtering and subsetting, equivalent to open-source R. All of these functions can be used in conjunction with open-source R capabilities. The TeradataAsterR package is compatible with RStudio and R Console. Open-source R and R packages embedded in the Aster Database. These embedded R packages allow users to deploy custom R scripts in a secure, parallel-processing environment using the Teradata Aster SQL-MapReduce Stream Module, while the data remains in the database. This approach provides both data integrity and computational efficiency benefits. Aster R automatically handles all tasks associated with parallel computations in a way that is transparent to the user. Users can develop an R script in the Aster R package on their client machine, and then install the final script onto the Aster Database. The script executes directly in the database to exploit the computing power of the Aster Database. Experienced users can build their own MapReduce functions using R packages for use in the Aster Database. The TeradataAsterR Package The TeradataAsterR package runs on the client system and is designed primarily for data management and exploration. The TeradataAsterR package includes over 100 functions organized into four functional areas: Utility and database management functions Teradata Aster R User Guide Update 3 7

8 Chapter 1: Introduction to Aster R R Embedded in the Aster Database Data exploration and preparation functions R version of all Teradata Aster Analytics functions These functions automate data partitioning and parallel processing across the nodes of the cluster. R script runners These runners allow users to run R scripts in the Aster Database, without installing open-source R in the database. Virtual Objects The TeradataAsterR package also includes a set of R data objects called virtual objects. Users can use Aster R commands to access and manipulate these virtual objects to access data in the Aster Database. The look and feel is very similar to that of interacting with a regular data frame in R. For example, Aster R provides an R object called virtual data frame that represents an Aster table, view, or SQL query. Aster R commands that access or manipulate a virtual data frame are translated to equivalent SQL or Teradata Aster SQL-MapReduce queries and are executed in the Aster Database using RODBC. R Embedded in the Aster Database Open-source R is designed to operate in a single-thread environment on data stored in the local system s memory. Because of this design, R fails with data too large to fit in memory. The amount of memory depends on the specific system configuration and the actual memory available at a given point in time. This limitation is exacerbated by the call-by-value semantics of an R execution, which leads to many copies of data being created in memory as data flows from one function to another. Data scientists and statisticians using R routinely analyze large data stored in relational databases. In most cases, the only option available for data scientists to analyze data stored in a relational database is to download the data into an R environment. This leads to a number of problems, including time-consuming data extraction from (and export to) relational databases. This typically prohibits interactive data analysis, unnecessarily duplicates data storage in the organization, and requires sampling or a system with large amounts of memory and storage to run R and process large amounts of data. The Aster R product addresses these challenges by making in-database execution of R possible. Executing R within the Aster Database eliminates the need to transfer data between the database and the R client. It also allows Aster R users to take advantage of parallel computation across many nodes and automatic scaling made possible by the Aster Database. 8 Teradata Aster R User Guide Update 3

9 CHAPTER 2 Installing Aster R Installation Overview The Aster R product consists of a number of components, and requires a number of steps to install. Some of the steps involve installing software onto a Linux cluster. In case the Aster R end user does not have such experience, or does not have the required permissions to carry out the tasks, assistance from Aster database administrators or IT personnel may be required to complete the installation procedures listed in this chapter. If the Aster cluster does not have direct access to the Internet, all R packages and required supporting operating system files must be downloaded onto a separate computer and then copied to the queen node of the Aster Database cluster. When R has direct access to an external R package repository such as CRAN, R handles package dependencies automatically - when a package is installed, all the additional required packages are installed as well. If the Aster cluster does not have direct access to the Internet, these dependencies are not resolved automatically, and the Aster R end user or database administrator must ensure that the dependencies are resolved. Teradata Aster provides a bundle that contains the most commonly used R packages, their dependencies, and the supporting files. Teradata recommends users install this bundle. Use Cases Aster R provides R capabilities in a couple of different ways. The required installation steps depend on how Aster R will be used. Use an R-like interface to the Aster Database for data exploration. The user installs the TeradataAsterR package on a client system connected to an Aster Database, and uses the client computer to query and access the Aster Database. This procedure is described in Installing Aster R on the Client. Run R scripts or execute R code directly on the Aster cluster. The user installs open-source R onto the Aster cluster. This procedure is described in Installing Open-Source R on the Aster Cluster. Because the Aster cluster typically does not have access to the Internet, the procedure to install R packages on the Aster cluster is different from that when using R on a personal laptop. Refer to Installing the Bundle of Commonly Used R Packages and Installing Additional Packages on the Cluster for information on how to install additional R packages on the Aster cluster. Teradata Aster R User Guide Update 3 9

10 Chapter 2: Installing Aster R Installing TeradataAsterR Package on the Client Installing TeradataAsterR Package on the Client System Requirements on the Client A supported version of open-source R must be installed on the client system. RStudio is optional. The supported version of open-source R is Open-source R can be downloaded from The supported versions of RStudio are: For Windows 64-bit and 32-bit platforms: RStudio version For Linux 64-bit platform: RStudio version , which can be downloaded from support.rstudio.com/hc/en-us/articles/ older-versions-of-rstudio. For Linux 32-bit platform: RStudio is not supported. Note: R Console is supported for this platform. In addition, the TeradataAsterR package requires that the following software be installed on the Aster cluster: Aster Database, version 6.20 (AD 6.20) or higher Aster Analytics, version (AA ) Note: Refer to the System Requirements on the Cluster for more details. Installing TeradataAsterR Package on the Client 1. Obtaining the TeradataAsterR Package 2. Setting Up the Aster ODBC Driver 3. Installing the TeradataAsterR package Two options available: Installing TeradataAsterR Package from RStudio Installing TeradataAsterR Package from R Console 4. Connecting to the Aster Database Obtaining the TeradataAsterR Package 1. Connect to the Teradata Developer Exchange website at 2. Click on the AsterR link next to the Aster Analytics Software link. Note: Contact a Teradata account manager if the link is not working on this web site. 3. Click on the package name for the operating system running on the client. 4. Log in to Teradata Developer Exchange, if prompted. 10 Teradata Aster R User Guide Update 3

11 5. Accept the End User License Agreement. 6. Download the package, and copy it to the client computer. Chapter 2: Installing Aster R Installing TeradataAsterR Package on the Client Setting up the Aster ODBC Driver Refer to the instructions of Installing and Configuring ODBC in the Teradata Aster Client Guide for information on obtaining, installing, and configuring Aster ODBC Driver on the client. For Windows platforms: refer to steps to Install ODBC on Windows. Note: Installing any ODBC driver on a Windows system may conflict with security protections. Address these items before proceeding: Users must have administrator privileges. Users may register the ODBC driver as a trusted application with any virus protection programs that are running; or temporarily turn off the virus protection program. For Linux platforms: refer to the steps to Install ODBC on Linux, Solaris, or Mac OS. Installing the TeradataAsterR Package Install the TeradataAsterR package using either of the following R interfaces: Install from RStudio Install from R Console Installing the TeradataAsterR Package from RStudio Users with RStudio installed on the client system can install the required packages using the Install Packages command from the Tools menu. 1. In the RStudio window, from the Tools menu, select Install Packages. 2. In the Install Packages window, select Package Archive File (.zip;.tar.gz) from the Install from drop-down menu. 3. Click Browse... to open the Select Package Archive window. 4. Locate and highlight the TeradataAsterR package, and click Open. 5. Click Install. 6. Repeat Step 1 to open the Install Packages window. 7. Select Repository (CRAN, CRANextra) from the Install from drop-down menu. 8. In the Packages field, type RODBC. 9. Click Install. Installing the TeradataAsterR Package from R Console Users can install the TeradataAsterR package from R Console. Use the commands here, and change the <path_to_package> and package name based on the actual file path and package name. Teradata Aster R User Guide Update 3 11

12 Chapter 2: Installing Aster R Installing TeradataAsterR Package on the Client Operating System Windows Command install.packages("<path_to_package>/ TeradataAsterR_ zip") install.packages("rodbc") Linux x86_64 install.packages("<path_to_package>/ TeradataAsterR_ _R_x86_64-unknown-linux-gnu.tar.gz") install.packages("rodbc") Linux i686 install.packages("<path_to_package>/ TeradataAsterR_ _R_i686-unknown-linux-gnu.tar.gz") install.packages("rodbc") Connecting to the Aster Database After Setting Up the Aster ODBC Driver and Installing the TeradataAsterR package on the client system, users can load the TeradataAsterR package, and use the ta.connect() function to establish a connection to the Aster Database from the R interface on the client system. 1. Load the TeradataAsterR package: > library(teradataasterr) 2. Use ta.connect to establish a connection to the Aster Database: > AsterR <- ta.connect("asterdb") The ta.connect() function returns an RODBC connection to the specified Aster Database. This object is assigned to a global variable "taconnection", which is used by default for other function calls unless the user provides an explicit RODBC connection object in the function call. The syntax of ta.connect(): ta.connect(dsn, uid="", pwd="", database="", dtype="odbc") where: dsn uid pwd database dtype For example: Data Source Name (DSN) for RODBC/ODBC connection. User Id. If not supplied, the user Id from the DSN is used. Password. If not supplied, the password from the DSN is used. Aster Database to connect to. If not supplied, the database name from the DSN is used. Driver type. The default driver is ODBC. Only ODBC is currently supported. ta.connect("asterdb") 12 Teradata Aster R User Guide Update 3

13 ta.connect("asterdb", uid="db_superuser", pwd="db_superuser", database="beehive", dtype="odbc") Postrequisite Confirming Worker Nodes When connecting to the Aster Database using ta.connect(), each worker node is registered and assigned a unique nodeid. This is how the worker nodes are made available for parallel processing. Confirm that all expected worker nodes are registered using ta.worker.nodenames(). Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Installing Open-Source R on the Aster Cluster When installing R and associated packages on the Aster cluster, they are installed in a "sandbox" area on each node in the cluster. To install R and its packages, users must first create this sandbox on the queen node, and then temporarily install a copy of the operating system with the same version as that is running on the queen node. After completing the R installation on the queen node, use Aster Database commands to synchronize the R installation across all nodes in the cluster. The operating system ISO images, RPM files or both can then be deleted from the sandbox. Note: The version of R installed on the Aster cluster and on the client must be the same in order to ensure proper operation of the Aster R package software. Note: The procedures in this section use ncli commands. Refer to the Teradata Aster Database User Guide for Aster Appliances and the Teradata Aster Database User Guide for Commodity Hardware for detailed information on ncli commands. Persistent Metadata This topic applies only to systems running a supported version of the Teradata Aster Execution Engine (for example, AX ). All R packages and the dependent OS packages that are installed and upgraded as specified in this chapter will be persistent across an Aster Execution Engine restart. Refer to the Teradata Aster Execution Engine Aster Instance User Guide for more information about objects that are persistable across an Aster instance restart or failover by using a checkpoint, and objects that are not persistable across an Aster instance restart or failover. System Requirements on the Cluster To install R on the cluster, the minimum requirements for the software environment are: Operating system: For Aster Appliance: a supported version of the Teradata SUSE Linux Enterprise Server (TD SLES) operating system is required. Teradata Aster R User Guide Update 3 13

14 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster For Commodity Hardware: a supported version of the Red Hat Enterprise Linux operating system (RHEL) or the SUSE Linux Enterprise Server (SLES) operating system is required. Note: Teradata recommends using SLES 12 SP2 or RHEL 7.3, with R 3.4.1, to match the R installed on the client side for full functionality. SLES 11 SP2 or RHEL 6, with R 3.1.2, is workable, but may not support the delta functions found in R that are not in R Database: Aster Database, version 6.20 (AD 6.20) or higher Aster Execution Engine (AX ) or higher Note: Only AD supports SLES 12 SP2. AD or higher supports RHEL 7.3. Aster Client, version 6.20 (AC 6.20) or higher Aster ODBC Driver (version AC 6.20 or higher) Prerequisites for Installing R on the Cluster The following files are required to install R on the cluster: ISO image files corresponding to the operating system on the cluster. R-specific RPM packages containing open-source R and other required software. Users may install additional R packages from either Teradata or other sources. Teradata provides a bundle of the most frequently used R packages and their dependencies at Instructions for installing this bundle are provided in Installing the Bundle of Commonly Used R Packages. To install additional R packages from other sources, follow instructions in Installing Additional Packages on the Cluster. Installing R on the Aster Cluster Running SLES Obtaining SLES ISO Image Files Contact system or network administrator to obtain the SLES and SLES SDK ISO image files. Request the same version of the SLES as the one running on the queen node. Alternatively, download the SUSE Linux Enterprise Server (SLE-12-SP2-Server-DVD-x86_64-GM- DVD1.iso) and SUSE Linux Enterprise Software Development Kit (SLE-12-SP2-SDK-DVD-x86_64-GM- DVD1.iso) from the SUSE website to the machine that has access to the Internet. 14 Teradata Aster R User Guide Update 3

15 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Note: Download and use the SLES-11-SP2-DVD-x86_64-GM-DVD1.iso and the SLE-11-SP2-SDK-DVDx86_64-GM-DVD1.iso image files, if installing on a system running SLES 11. Note: A Suse/Novell account is required to log on to the website to download the ISO image files. Create an account if the user does not already have one. Note: Do not download the most recent version of the files. Use the site and search for the desired ISO image files. Obtaining R RPM Files for SLES 1. Download the required R and R dependency RPM files. This procedure must be performed on a machine that has access to the Internet. a) Go to and log on to the website. b) Locate the R-3.4.1_sles12_rpms.tar.gz link on the web page and download the tarball by clicking it. Note: Use the R-3.1.2_sles11_rpms.tar.gz tarball, if installing on a system running SLES 11. c) Extract the tarball. 2. On the queen node of the Aster Database cluster, if the /var/opt/teradata/sles-r directory does not already exist, create the directory by running this command: mkdir -p /var/opt/teradata/sles-r 3. Copy the RPM files from the machine on which they were downloaded to the /var/opt/teradata/ sles-r directory on the queen node of the Aster Database cluster. Mounting SLES ISO Image Files Perform these steps on the queen node to mount the SLES ISO image files. 1. Copy the SLES ISO image files into the /var/opt/teradata directory: scp <path_to_sles-iso-image-file> /var/opt/teradata/ 2. Create the /var/opt/teradata/sles and the /var/opt/teradata/sles-sdk directories: mkdir -p /var/opt/teradata/sles mkdir -p /var/opt/teradata/sles-sdk 3. Change to the /var/opt/teradata directory: cd /var/opt/teradata 4. Mount the SLES ISO image files: mount -o loop SLE-12-SP2-Server-DVD-x86_64-GM-DVD1.iso /var/opt/teradata/ sles mount -o loop SLE-12-SP2-SDK-DVD-x86_64-GM-DVD1.iso /var/opt/teradata/slessdk Teradata Aster R User Guide Update 3 15

16 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Installing R on SLES 1. Run this command (one single command) on the queen node to install R: ncli apm install R --usedefaultrrepo=false --repo=sles,file:///var/opt/ teradata/sles --repo=sles-sdk,file:///var/opt/teradata/sles-sdk --repo=slesr1,file:///var/opt/teradata/sles-r 2. Run this command on the queen node to confirm that R is successfully installed: ncli apm show R --localconfig The output should show the R version installed on the queen node. Preparing the Cluster for Aster R Runner Functions Teradata recommends users to follow these steps to optimize performance when using the Aster R runner functions. These steps create and mount the directory home/extensibility/stream on Aster cluster. During an Aster R session, virtual object files created using the R runner functions are saved in this directory. When the Aster R session ends, these objects are deleted. For more information on Aster R runner functions and Aster virtual objects, see Using Aster R Runner Functions to Run R Functions in the Aster Database. Note: Run the following commands on the queen node. 1. ncli node runonother "mkdir /data/stream" 2. ncli node runonother "chown -R extensibility:extensibility /data/stream" 3. ncli node runonother "mkdir /home/extensibility/stream" 4. ncli node runonother "chown -R extensibility:extensibility /home/ extensibility/stream" 5. ncli node runonother "mount --bind /data/stream /home/extensibility/stream" Syncing R across All Workers after Installation The command in the installation step installs R only on the queen node. 1. To sync R across all workers in the cluster: a) On systems running a supported version of the Aster Database (AD and later) or a supported version of the Aster Execution Engine (for example, AX ), issue this command: ncli apm administer R --synchronize b) On systems running a supported version of the Aster Database (AD and earlier), issue these commands: ncli system softrestart ncli system activate 2. To confirm that R is successfully installed on all worker nodes in the cluster, run this command on the queen node: ncli apm show R --localconfig 16 Teradata Aster R User Guide Update 3

17 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster The command should run without any errors, and the output should show the R version installed on the queen node and the worker nodes. All nodes should have the same R version. The R and R packages on Aster are installed in the /opt/aster/third-party/r directory on the queen and worker nodes in the Aster Database cluster. Installing R on the Aster Cluster Running RHEL Prerequisite The yum and createrepo commands must work correctly on the queen node of the cluster where R will be installed. To verify that the commands are working correctly, confirm that an error is not returned after issuing each of these commands on the queen node: yum --version createrepo -version Obtaining RHEL ISO Image File Contact system or network administrator to obtain the RHEL ISO image file. Ensure that the requested RHEL ISO file is of the same version as that of the main RHEL operating system running on the queen node. Obtaining R RPM Files for RHEL 1. Download the required R and R dependency RPM files. This procedure must be performed on a machine that has access to the Internet. a) Go to and log on to the website. b) Locate the R-3.4.1_rhel7_rpms.tar.gz link on the web page and download the tarball by clicking it. Note: Use the R-3.1.2_rhel6_rpms.tar.gz tarball, if installing on a system running RHEL 6.7/6.8. c) Extract the tarball. 2. On the queen node of the Aster Database cluster, if the /var/opt/teradata/rhel-r directory does not already exist, create the directory by running this command: mkdir -p /var/opt/teradata/rhel-r 3. Copy the RPM files from the machine where they were downloaded to the /var/opt/teradata/ rhel-r directory on the queen node of the Aster Database cluster. 4. On the queen node, run these commands to create a yum repository from all files copied in Step 3: cd /var/opt/teradata/rhel-r createrepo. Mounting RHEL ISO Image File Perform these steps on the queen node to mount the RHEL ISO image file. 1. Copy the RHEL ISO image file into the /var/opt/teradata directory: Teradata Aster R User Guide Update 3 17

18 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster scp <path_to_rhel-iso-image-file> /var/opt/teradata/ 2. Create the /var/opt/teradata/rhel directory: mkdir -p /var/opt/teradata/rhel 3. Change to the /var/opt/teradata directory: cd /var/opt/teradata 4. Mount the RHEL ISO image file: mount -o loop rhel-server-7.3-x86_64-dvd.iso /var/opt/teradata/rhel Note: Use the corresponding RHEL ISO image file, if installing on a system running RHEL 6.7/6.8. Installing R on RHEL 1. Run this command (one single command) on the queen node to install R: ncli apm install R --usedefaultrrepo=false --repo=rhel,file:///var/opt/ teradata/rhel --repo=rhel-r,file:///var/opt/teradata/rhel-r 2. Run this command on the queen node to confirm that R is successfully installed: ncli apm show R --localconfig The output should show the R version installed on the queen node. Preparing the Cluster for Aster R Runner Functions Teradata recommends users to follow these steps to optimize performance when using the Aster R runner functions. These steps create and mount the directory home/extensibility/stream on Aster cluster. During an Aster R session, virtual object files created using the R runner functions are saved in this directory. When the Aster R session ends, these objects are deleted. For more information on Aster R runner functions and Aster virtual objects, see Using Aster R Runner Functions to Run R Functions in the Aster Database. Note: Run the following commands on the queen node. 1. ncli node runonother "mkdir /data/stream" 2. ncli node runonother "chown -R extensibility:extensibility /data/stream" 3. ncli node runonother "mkdir /home/extensibility/stream" 4. ncli node runonother "chown -R extensibility:extensibility /home/ extensibility/stream" 5. ncli node runonother "mount --bind /data/stream /home/extensibility/stream" Syncing R across All Workers after Installation The command in the installation step installs R only on the queen node. 1. To sync R across all workers in the cluster: a) On systems running a supported version of the Aster Database (AD and later) or a supported version of the Aster Execution Engine (for example, AX ), issue this command: 18 Teradata Aster R User Guide Update 3

19 ncli apm administer R --synchronize b) On systems running a supported version of the Aster Database (AD and earlier), issue these commands: ncli system softrestart ncli system activate 2. To confirm that R is successfully installed on all worker nodes in the cluster, run this command on the queen node: ncli apm show R --localconfig Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster The command should run without any errors, and the output should show the R version installed on the queen node and the worker nodes. All nodes should have the same R version. The R and R packages on Aster are installed in the /opt/aster/third-party/r directory on the queen and worker nodes in the Aster Database cluster. Installing the Bundle of Commonly Used R Packages Teradata provides an automated bundle that installs 104 of the most commonly used R packages and all of the associated R and OS dependencies. Teradata recommends the use of the installer included in the bundle for installing this bundle. The bundle (including R packages, all of the associated R and OS dependencies, the installer and associated documentation) can be downloaded from Installation instructions are included in the downloaded bundle, but are provided here for convenience. Note: This procedure automatically takes care of synchronizing the worker nodes with the queen node after installing the packages. It is not necessary to issue a separate synchronization command. 1. Download the RPackagesBundle_<ReleaseVersion>.tar.gz. 2. Copy the downloaded file to the /var/opt/teradata/ directory on the queen node. 3. Log in to the queen node as root. 4. Change to the /var/opt/teradata directory. cd /var/opt/teradata/ 5. Unzip the tarball: tar -xvf RPackagesBundle_<ReleaseVersion>.tar.gz 6. Change to the RPackagesBundle directory: cd RPackagesBundle 7. Run the install.sh script to install R packages:./install.sh Installing Additional Packages on the Cluster This section describes how to install additional packages that users may need. Teradata Aster R User Guide Update 3 19

20 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Before installing additional packages, user should install the bundle provided by Teradata that includes the most commonly used R packages and all of their associated dependencies (refer to Installing Bundle of Commonly Used R Packages). Creating a Repository for R and OS Packages The R package installer internally calls the Yum (for RHEL) or Zypper (for SLES) package installer. Yum and Zypper require a repository that holds the OS packages to be installed. Before installing additional R packages or related OS executables, users must create this repository on the queen node of the Aster cluster. Use these steps to create a repository and then install the packages. 1. Log in to the queen node of the Aster cluster as root. 2. Execute these commands: mkdir /var/opt/teradata/rrepo ncli apm administer R --setuprepo=rrepo,file:///var/opt/teradata/rrepo 3. Copy the R packages or OS executables that need to be installed on the cluster into the repository. Removing a Permanent Repository Run this command to remove a permanent repository: # ncli apm administer R --removerepo=<reponame> Note: This repository is available only in the R sandbox area. If R is uninstalled, the repository is also deleted. Obtaining R Packages If the Aster cluster does not have direct Internet access, R packages must be downloaded from the Internet to a separate machine and then copied to the queen node of the Aster cluster. There are several options to perform this task. Download R packages from the CRAN website at This is the most common source for R packages and documentation. When downloading R packages manually from the CRAN website, follow the steps below to make sure that all dependencies are downloaded as well. Go to the CRAN website, navigate to packages and then to Table of available packages, sorted by name. When the user click on a package name, a screen appears with links to all of the package documentation, including a list of required R package dependencies (on the line Imports, as shown here). 20 Teradata Aster R User Guide Update 3

21 Another method is to write an R script that connects to CRAN, queries the R package repository, resolves package dependency issues, and then downloads all the required R packages to a local folder. An example script is shown below. Note that this script does not provide the required ordering of packages when inputting more than one package at a time into the "target" object. ### Step 1: Change this to create initial package list. Enter package names between quotes # separate package names with commas and be careful with punctuation targets <- c("r_packagename_1", " r_packagename_2") ### Step 2: Change Folder variable to enter destination file directory # Please note that with R the direction of the slash is the reverse of the Windows norm # Windows uses backward slash "C:\Users\..." while R uses forward slash "C:/ Users/..." folder <- "C:/Documents/Aster/Installation/R/packages" ### Step 3: Establish function getpackages <- function(packs){ packages <- unlist( tools::package_dependencies(packs, available.packages(), which=c("depends", "Imports", "LinkingTo" ), recursive=true) ) packages <- union(packs, packages) packages } packages <- getpackages(targets) ### Step 4: Execute function download.packages(packages, destdir=folder,type="source") Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Note: In almost all cases, the R packages required by Aster are the same as the ones installed in a Windows installation of R. R packages downloaded from CRAN have a tar.gz extension. R packages downloaded from other sources will probably have a.zip extension. Linux-oriented.rpm files are not used with Aster R. Installing R Packages This procedure lists steps to install R packages, and assumes that the user has created a repository in the R sandbox area as described in Creating a Repository for R and OS Packages. 1. Log in to the queen node of the Aster cluster as root. 2. Copy the R packages into the repository created (/var/opt/teradata/rrepo). 3. Change to the repository directory. cd /var/opt/teradata/rrepo 4. Reset the security settings for the rrepo folder. Teradata Aster R User Guide Update 3 21

22 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster chmod -R 755 * 5. Install R packages. ncli apm install R --packages=<rpackage1filename>,<rpackage2filename>,... For example: ncli apm install R --packages=lpsolve_ tar.gz ncli apm install R --packages=lpsolve_ tar.gz,sampling_2.6.tar.gz Note: Some R packages may have a different internal structure. If the above command does not work, try using the package name instead of the filename, as shown here: ncli apm install R --packages=<rpackage1packagename> For example: ncli apm install R --packages=lpsolve Troubleshooting R Package Installation Scenario One: If the installation fails, review the error message and log files to see if there are any dependencies that may be missing from the repository. The log files for the installation process are located at: /primary/logs/ ncli-[time stamp].log. To resolve installation failures due to dependencies on other R packages, download and install the dependencies first, and then install the R package that failed previously. Scenario Two: If the queen node enters a failed state after the user installing the R packages and synchronizing the worker nodes, R may not be properly synced due to a permission issue. To correct the issue, run this command: find /opt/aster/third-party/r/! -type l -path '*/R/proc' -prune -o -path '*/R/ dev' -prune -o -path '*/R/home' -prune -o -path '*/R/root' -prune -o! -path '*/R/etc/localtime'! -path '*/R/etc/nsswitch.conf'! -path '*/R/etc/group'! - path '*/R/etc/passwd'! -path '*/R/etc/resolv.conf'! -path '*/R/etc/shadow'! -path '*/R/etc/gshadow'! -user beehive! -perm -o+r -exec chown beehive:beehive {}; Installing OS Packages inside R Sandbox Area Some R packages rely on specific OS packages to work properly. In order for these R packages to work in the sandbox area, all OS dependencies must also be installed in the sandbox. This procedure assumes that the user has created a repository as described in Creating a Repository for R and OS Packages, and that the required packages have been copied into the repository. Use this command to install OS packages inside the R sandbox area: ncli apm install R --rpmpackages=<package1>,<package2>,...[-- repo=<reponame1>,<repourl1>] [--repo=<reponame2>,<repourl2>]... The following examples assume that the packages "tar", "less", and "zip" are in the repository: ncli apm install R --rpmpackages=tar 22 Teradata Aster R User Guide Update 3

23 ncli apm install R --rpmpackages=less,zip Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Commands to Manage Packages on the Cluster This section lists common commands used to manage R packages on the Aster cluster. Action Display all currently installed R packages Check installation status of an R package Check status of an OS package installed in the R sandbox area Remove R packages Upgrade specific R packages Command ncli apm show R --packages ncli apm show R --packages=<rpackage1>,<rpackage2>,... Example: ncli apm show R --packages=tree ncli apm show R --packages=tree,lpsolve ncli apm show R --rpmpackage=<os_package> Example: ncli apm show R --rpmpackage=tar ncli apm remove R --packages=<rpackage1>,<rpackage2>,... Example: ncli apm remove R --packages=tree ncli apm remove R --packages=tree,lpsolve ncli apm install R --packages=<pkg1>[,<pkg2>] Note: Packages will only be upgraded if the packages specified in the command are more recent than the version already installed. Upgrade OS packages installed in the R sandbox area ncli apm upgrade R --rpmpackages=<package1>,<package2>,... Example: ncli apm upgrade R --rpmpackages=tar Note: The new version of the OS package must be available in the repo directory. Environment Variables on the R Client Environment Variables In case of need, users can adjust the environment variables in this table to improve performance. For most use cases, the default setting is optimal. Environment Variable MAX_NODE_ID Description This variable controls the random allocation of IDs to vworkers. Default 99 Teradata Aster R User Guide Update 3 23

24 Chapter 2: Installing Aster R Installing Open-Source R on the Aster Cluster Environment Variable SERVER_STREAM _DIR STREAM_READ_B ATCH_CNT STREAM_MEMOR Y_LIMIT_MB Description It must not be smaller than the total number of vworkers in the cluster, which is the number of nodes multiplied by the number of vworkers per node. And it has a range from 50 to 999. The default setting is large enough for standard Aster Database configurations of 16 worker nodes. Adjusting the value for this variable may improve performance for small clusters when establishing a connection to the database using the AsterR package, and is required if the cluster has more than 16 worker nodes. This variable sets the directory on the Aster cluster to store virtual object files. This variable controls the number of records to read as a batch for stream processing. The default is all records read at once. Adjusting the value for this variable if the input data stream is larger than the worker node memory. Note: When this variable is set to a non-default value, the operation is performed independently for each batch of stream data. This is not recommended for aggregate operations such as training a model. This variable controls the amount of memory allocated to the stream for the execution of a R function using aa.apply and aa.tapply. If the system reports memory allocation errors, try increasing the value of this variable. Default /home/ extensibility/ stream Commands to display and change the environment variables To display the value of an environment variable: sys.getenv( STREAM_MEMORY_LIMIT_MB ) To change the value of an environment variable: sys.getenv(stream_memory_limit_mb = 20000) 24 Teradata Aster R User Guide Update 3

25 CHAPTER 3 Working with Tables and Data Frames in Aster R Working with Tables and Data Frames in Aster R This chapter describes the Aster R functions available for working with tables, including transferring data between Aster virtual data frames and R data frames, and reviewing and manipulating data. These functions allow users to use R-style functions to examine and manipulate data stored in an Aster Database. Aster R uses the concept of virtual object. These objects represent data stored in the Aster Database, and can be manipulated from the R command line using Aster R functions. Virtual objects can be converted to regular R objects and vice versa. For details about functions, refer to the Aster R Function Reference or to the inline help. Note: The examples in this chapter use R dataset "women", which can be found in the R package "datasets". Storing, Exploring and Transforming Data Creating Tables and Virtual Data Frames Creating Table in the Aster Database Users can use the function ta.create() to create a table in the Aster Database based on an R data frame. The function takes the provided arguments, and generates a SQL query that creates a table in the Aster Database. Users can start with verifying the content in the R dataset "women", available in the R package "dataset". > women height weight Teradata Aster R User Guide Update 3 25

26 Chapter 3: Working with Tables and Data Frames in Aster R Storing, Exploring and Transforming Data > class(women) [1] "data.frame" This example uses the function ta.create() to create a table named "women_ht_wt" in the "public" schema of the Aster Database. The table contains the same data as the "women" dataset, and can be manipulated using Teradata Aster SQL-MapReduce commands. ta.create(women, table="women_ht_wt", schemaname="public", tabletype="dimension", row.names=false, coltypes=null ) Creating Virtual Data Frame in the Aster Database Users can use the function ta.data.frame() to create a virtual data frame from an existing table or a SELECT query. This example creates a virtual data frame "tadf_women" from the existing Aster Database table "women_ht_wt". > tadf_women<-ta.data.frame('women_ht_wt') > class(tadf_women) [1] "ta.data.frame" Users can manipulate the virtual data frame "tadf_women" using Aster R functions. In addition, the functions ta.vector() and ta.factor() are available to represent a single column in a table. And the functions as.ta.vector() and as.ta.factor() are available to convert an existing single-column virtual data frame to a ta.vector or ta.factor object. Exploring Data Aster R provides functions to explore virtual data frames. This table lists brief descriptions of some of these functions. Function ta.head() ta.tail() ta.show() ta.dim() Description Returns the first 6 rows of the virtual data frame. Returns the last 6 rows of the virtual data frame. Returns all rows of the virtual data frame. Returns a numeric vector (number of rows, number of columns) showing the dimensions of the virtual data frame. 26 Teradata Aster R User Guide Update 3

27 Chapter 3: Working with Tables and Data Frames in Aster R Storing, Exploring and Transforming Data Function ta.length() ta.names() ta.ncol() ta.nrow() ta.colnames() ta.dimnames() Description Returns the length of the virtual object. For objects of class ta.vector or ta.factor, the length is the number of rows. For objects of class ta.data.frame, the length is the number of columns. For objects of class ta.list, the length is the number of elements. Returns the names attribute of the virtual data frame as a character vector. Returns the number of columns of the virtual data frame. Returns the number of rows of the virtual data frame. Can be used to return or alter the column names of the virtual data frame. Returns a character vector with the dimension names. Transforming Data Aster R provides functions to transform data. This table lists brief descriptions of some of these functions. Function Combining data ta.merge() ta.join() ta.cbind() ta.rbind() Ordering data ta.order() Subsetting data ta.subset() Recoding ta.recode() Transforming columns ta.transform() ta.with() Description Returns a virtual data frame based on a JOIN of two input virtual data frames. Returns a virtual data frame based on a JOIN of two input virtual objects using one or more join keys. Returns a virtual data frame combining the columns of the two input data frames. The input data frames must have a row_names column and the same number of rows. Returns a virtual data frame that is a UNION of the two input data frames. Returns a virtual data frame ordered as specified by the input parameters. Returns a virtual data frame based on the subset criteria specified by the input parameters. Returns a virtual data frame with a specific column updated as specified by the input parameters. Returns a virtual data frame with one or more columns updated as specified by the input parameters. Returns the value of a specific expression based on the specified input data frame. Teradata Aster R User Guide Update 3 27

28 Chapter 3: Working with Tables and Data Frames in Aster R Loading and Extracting Data Function ta.within() Description Returns a virtual data frame modified by evaluating the specified expression. Converting between Aster R Virtual Objects and Regular R Objects Aster R provides functions to automatically convert between virtual and regular R objects. This table lists brief descriptions of these functions. Function ta.data.frame( ) as.ta.data.fra me() as.data.frame( ) as.ta.factor() as.ta.vector() as.vector() Description Creates a virtual data frame based on an underlying SELECT query. The SELECT query is created based on the input arguments. Converts an existing R object or Aster virtual object to an Aster virtual data frame. Converts a table in the Aster Database to a virtual object that can be accessed using Aster R functions. Converts an R object or Aster R virtual object to an Aster R virtual factor. The object to be converted must have a single column of type character. Converts an R object or Aster R virtual object to an Aster R virtual vector. The object to be converted must have a single column. Converts a virtual vector to a regular R vector. Loading and Extracting Data Loading Data to Aster Users can use the function ta.push() to load data from R into an existing Aster virtual data frame. For example: > class(tadf_women) [1] "ta.data.frame" > new_row<-cbind(73,169) > new_row [,1] [,2] [1,] > ta.push(tadf_women, new_row) > tadf_women height weight Teradata Aster R User Guide Update 3

29 Warning message: In ta.show(tadf, maxrows) : Printing rows in random order since base table/view is neither ordered nor have row_names column. Users can use the function ta.load.csv() to load data from a file in.csv format to an Aster virtual data frame. For example: > ta.load.csv(tadf_women, file="new_row_57_112.csv", header=false) > tadf_women height weight Warning message: In ta.show(tadf, maxrows) : Printing rows in random order since base table/view is neither ordered nor have row_names column. Note: The "Warning message" in the output is an explanatory note to the user. Chapter 3: Working with Tables and Data Frames in Aster R Loading and Extracting Data Exporting Data from Aster Users can use the function ta.pull() to export data from an Aster virtual object to an R object. The function returns an R data frame. By default, character strings including factor levels are encoded as UTF-8. Teradata Aster R User Guide Update 3 29

30 Chapter 3: Working with Tables and Data Frames in Aster R Loading and Extracting Data For example: > R_women<-ta.pull(tadf_women) > class(r_women) [1] "data.frame" Users can use the function ta.write.csv() to export data from an Aster virtual object to a file in.csv format. For example: > class(tadf_women) [1] "ta.data.frame" > ta.write.csv(tadf_women, file="women.csv") 30 Teradata Aster R User Guide Update 3

31 CHAPTER 4 Using Aster Analytics Functions with Aster R Using Aster Analytics Functions with Aster R Aster R provides an R interface to all of the Aster Analytics pre-built functions. This chapter shows some examples of how the Aster R package allows users to run Aster Analytics functions through an R interface, with no SQL commands required. Using the Naïve Bayes Model with Aster R This section uses the datasets "Pima.tr" and "Pima.te" found in the R package "MASS". These datasets are predefined training and test subsets from a dataset consisting of seven biomarker measurements from 532 women of Pima Indian heritage. These women were also tested for the presence of diabetes. An eighth column indicates whether diabetes was present or not. In this example, users will build a Naïve Bayes classifier based on the training dataset, apply it to classify the rows in the test dataset, and create a confusion matrix to evaluate the model's performance. 1. Create tables in the Aster Database to hold the data. ta.create(pima.tr, table="pima_train", schemaname="public", tabletype="dimension", row.names=true, coltypes=null ) ta.create(pima.te, table="pima_test", schemaname="public", tabletype="dimension", row.names=true, coltypes=null ) 2. Create virtual data frames. tadf_pima.tr<-ta.data.frame('pima_train') tadf_pima.te<-ta.data.frame('pima_test') 3. Create the Naïve Bayes model using the training dataset. nbmodel<-aa.naivebayes.train( formula = (type ~ npreg + glu + bp + skin + bmi + ped + age), Teradata Aster R User Guide Update 3 31

32 Chapter 4: Using Aster Analytics Functions with Aster R Using the Naïve Bayes Model with Aster R data = tadf_pima.tr ) 4. Run the model on the test dataset. pred<-aa.naivebayes.predict( object = nbmodel, newdata = tadf_pima.te, id.col = "row_names" ) [[1]] row_names prediction loglik_no loglik_yes 1 1 Yes No No No Yes Yes No No Create a data frame containing the "prediction" column from the output of Step 4 and the "type" column from the input "Pima_test" table. predicted_values<-as.ta.data.frame(pred[[1]]) joined_table<-ta.join(predicted_values, tadf_pima.te, type="inner", by="row_names") ConfMatInput<-joined_table[,c("prediction","type")] > ConfMatInput prediction type 1 Yes Yes 2 No No 3 No No 4 No Yes 5 Yes Yes 6 Yes Yes 7 No Yes 8 No No 9 Yes No 6. Review the confusion matrix to analyze the model s performance. a) Use the function aa.confusion.matrix() to create tables in the Aster Database. aa.confusion.matrix(confmatinput, reference = 'type', prediction = 'prediction', output.tablename.prefix = "NBexample" ) The function creates three tables in the Aster Database: "nbexample_1", "nbexample_2", and "nbexample_3". b) Use the function ta.pull() to bring the tables into the R environment. nb1<-ta.pull("nbexample_1") nb2<-ta.pull("nbexample_2") nb3<-ta.pull("nbexample_3") 32 Teradata Aster R User Guide Update 3

33 c) Examine the results. > nb1 observation/predict No Yes 1 No Yes > nb2 key value 1 Accuracy % CI (0.7062, ) 3 Null Error Rate P-Value [Acc > NIR] Kappa Mcnemar Test P-Value > nb3 measure No Yes 1 Sensitivity Specificity Pos Pred Value Neg Pred Value Prevalence Detection Rate Detection Prevalence Balanced Accuracy Chapter 4: Using Aster Analytics Functions with Aster R Using the Random Forest Model with Aster R Using the Random Forest Model with Aster R This section uses the dataset "fgl" found in the R package "MASS". This dataset includes nine different measurements on 214 samples of different types of glass. A tenth column indicates the type of glass, classifying the samples into one of six types. This section also illustrates the use of the ta.push() function to transfer data from R into the Aster Database. 1. Add an "id" column to the data frame. fgl_with_rowids<-cbind(rownames(fgl), fgl) 2. Create an empty data frame. id<-integer() RI<-numeric() Na<-numeric() Mg<-numeric() Al<-numeric() Si<-numeric() K<-numeric() Ca<-numeric() Ba<-numeric() Fe<-numeric() type<-character() glass_data<-data.frame(id,ri,na,mg,al,si,k,ca,ba,fe,type) Teradata Aster R User Guide Update 3 33

34 Chapter 4: Using Aster Analytics Functions with Aster R Using the Random Forest Model with Aster R 3. Use the empty data frame to create an empty Aster Database table with the same schema to hold the data. ta.create(glass_data, table="fgltmptable", schema="public", tabletype="dimension" ) 4. Create a virtual data frame. This virtual data frame enables users to use Aster R functions to access and manipulate the data. tadf_glass<-ta.data.frame("fgltmptable") 5. Use the function ta.push() to copy the data from the R data frame "fgl_with_rowids" to the virtual data frame "tadf_glass". tadf_glass<-ta.push(tadf_glass, fgl_with_rowids) ta.head(tadf_glass) id RI Na Mg Al Si K Ca Ba Fe type WinF WinF WinF WinF WinF WinF 6. The next step is to divide the data into training and test datasets. To ensure that the training set has representatives of each of the six types of glass, split the table by the "type" column, and then divide the observations for each type into training and test subsets. Then combine the training and test subsets for each type to create training and test subsets that cover the entire dataset. a) Use the function ta.split() to split the virtual data frame according to the glass type. glass_types<-ta.split(tadf_glass, "type") The output of the ta.split() function is a ta.list. b) Get each individual data frame from the ta.list "glass_types". Con<-glass_types[[1]] Head<-glass_types[[2]] Tabl<-glass_types[[3]] Veh<-glass_types[[4]] WinF<-glass_types[[5]] WinNF<-glass_types[[6]] 34 Teradata Aster R User Guide Update 3

35 c) Create the training and test subsets for each glass type, using 70% for the training subsets and 30% for the test subsets. Con_train_indices=sample(1:ta.nrow(Con), 0.7*ta.nrow(Con)) Con.test=Con[-Con_train_indices,] Con.train=Con[Con_train_indices,] Tabl_train_indices=sample(1:ta.nrow(Tabl), 0.7*ta.nrow(Tabl)) Tabl.test=Tabl[-Tabl_train_indices,] Tabl.train=Tabl[Tabl_train_indices,] Veh_train_indices=sample(1:ta.nrow(Veh), 0.7*ta.nrow(Veh)) Veh.test=Veh[-Veh_train_indices,] Veh.train=Veh[Veh_train_indices,] Head_train_indices=sample(1:ta.nrow(Head), 0.7*ta.nrow(Head)) Head.test=Head[-Head_train_indices,] Head.train=Head[Head_train_indices,] WinF_train_indices=sample(1:ta.nrow(WinF), 0.7*ta.nrow(WinF)) WinF.test=WinF[-WinF_train_indices,] WinF.train=WinF[WinF_train_indices,] WinNF_train_indices=sample(1:ta.nrow(WinNF), 0.7*ta.nrow(WinNF)) WinNF.test=WinNF[-WinNF_train_indices,] WinNF.train=WinNF[WinNF_train_indices,] d) Combine the training and test subsets to create training and test datasets that contain samples of each of the six glass types. fgl.test<-rbind(winnf.test, Con.test, Tabl.test, Veh.test, WinF.test, Head.test) fgl.train<-rbind(winnf.train, Con.train, Tabl.train, Veh.train, WinF.train, Head.train) 7. Create virtual data frames for the training and test datasets. tadf_test<-as.ta.data.frame(fgl.test) tadf_train<-as.ta.data.frame(fgl.train) 8. Create the Random Forest model using the training dataset. glass_rf_list<-aa.forest( formula = (type~ri+na+mg+al+si+k+ca+ba+fe), tree.type = "classification", data = tadf_train ) glass_rf_list_1<-aa.forest( formula = (type~ri+na+mg+al+si+k+ca+ba+fe), tree.type = "classification", data = tadf_train, ntree = 6, mtry = 3 ) 9. Predict on the test dataset. aa.forest.predict( object = glass_rf_list, newdata = tadf_test, Chapter 4: Using Aster Analytics Functions with Aster R Using the Random Forest Model with Aster R Teradata Aster R User Guide Update 3 35

36 Chapter 4: Using Aster Analytics Functions with Aster R Using Sentiment Extraction with Aster R id.column = "id" ) aa.forest.predict( object = glass_rf_list_1, newdata = tadf_test, id.column = "id" ) Using Sentiment Extraction with Aster R This example uses the Aster sentiment extraction function to evaluate and classify a set of restaurant reviews. The input data table "restaurant_reviews" table is listed here. id review_text 1 This restaurant was great. The food was amazing. Our waiter was excellent. The appetizers in particular were very creative and well-thought-out. 2 I really enjoyed my meal, and my daughter's steak was perfectly prepared. The chocolate torte was superb. 3 The service was terrible! The food was ok, but the bread was stale and the drinks were very weak. 4 Not a must-do. Perfectly adequate but nothing special for the price. 5 I can't recommend this place. Service was slow and unfriendly. Food so-so. 6 Definitely a good choice for a special occasion. Highly recommended! 1. Create a data frame from the input data table "restaurant_reviews". tadf_restaurant_reviews <- ta.data.frame('restaurant_reviews') This example uses the default values for many arguments, in particular for the object argument which specifies the source used to assign sentiment values to words. The default value for the object argument is a built-in dictionary based on the WordNet lexical database. 2. Call the sentiment extraction function. result <- aa.sentiment.extract( newdata=tadf_restaurant_reviews, text.column='review_text', accumulate=c('id') ) The output is shown here. 36 Teradata Aster R User Guide Update 3

37 Using Text Analysis with Aster R This section uses a log of vehicle complaints that have been categorized as crash-related or not crash-related. Users can use this log to build a Naïve Bayes Text Classifier model, and then apply the model to a new log data to predict if the complaint is associated with a crash. It is assumed that the training and test datasets are already in the Aster Database. This is the table containing the training dataset. Chapter 4: Using Aster Analytics Functions with Aster R Using Text Analysis with Aster R doc_id text_data category 1 consumer was driving approximately 45 mph hit a deer with the front bumper and then ran into an embankment head-on passenger's side air bag did deploy hit windshield and deployed outward. driver's side airbag cover opened but did not inflate it was still folded causing injuries. 2 when vehicle was involved in a crash totalling vehicle driver's side/ passenger's side air bags did not deploy. vehicle was making a left turn and was hit by a ford f350 traveling about 35 mph on the front passenger's side. driver hit his head-on the steering wheel. hurt his knee and received neck and back injuries. 3 consumer has experienced following problems; 1.) both lower ball joints wear out excessively; 2.) head gasket leaks; and 3.) cruise control would shut itself off while driving without foot pressing on brake pedal. 4 transfer case was repaired under recall. after the work was completed noise was heard intermittently. consumer took vehicle back to dealer. the dealer re-inspected vehicle and informed the owner that the driveshaft was hitting the transfer case. the manufacturer has been notified. 5 transmission would start to slip when traveling just 10mph. the rpms would be over 3 thousand. had vehicle checked at dealership & was informed transmission was stuck & that it's a factory defect almost blew up. also speedometer does not keep accurate speeds. if speed is increased, it would fail to work. this was referred to mechanic by manufacturer. 6 due to the defective ignition cable which burned the coil the vehicle stalled unexpectedly which could have resulted in a crash. also dealer replaced the r&r drive belts/speed control cable and performed vehicle tune up. please provide further information. 7 when switch is turned on windshield wipers would not work properly. would have to jiggle switch & then wipers would move. wipers do turn off/on by themselves. recall 97v consumer was driving in a rain storm when the windshield wipers stopped this happened periodically. crash crash no_crash no_crash no_crash no_crash no_crash no_crash Teradata Aster R User Guide Update 3 37

38 Chapter 4: Using Aster Analytics Functions with Aster R Using Text Analysis with Aster R doc_id text_data category 9 at miles transmission has malfunctioned and will not shift into first gear. repairs were made at owner's expense wants reimbursement. *ml 10 when truck was sitting on an incline it rolled on its own. manufacturer was aware of the problem. problem has not been corrected. the truck is owned by walnut hill recker manufactured in car engine raced while slowing to park. car lurched forward and crashed into a fence and a building. car had been in shop approximately one week prior to incident for high idle condition. 12 rear ended another vehicle at 65 to 70mph and neither driver's side or passenger's side airbags deployed. dealer has vehicle. 13 while vehicle was parked for an hour a fire started on the left side of the engine compartment. owners son smelled smoke owner saw fire coming from around drivers side front wheel. referenced in ea after vehicle was repaired under recall 99v ignition switch the airbag light stayed on. the dealer and the manufacturer has been notified. 15 electrical control module is shortening out causing the vehicle to stall. engine will become totally inoperative. consumer had to change alternator/ battery and starter and module replaced 4 times but defect still occurring cannot determine what is causing the problem. 16 at miles power steering broke off the housing pump causing total loss of power steering which also caused the vehicle to shut down. 17 on two occasions dual airbags did not deploy. consumer rearended another vehicle at approximately 50 mph and at 80 mph hit a truck head-on upon impact air bags did not deploy. driver sustained injuries. dealer did not determine why air bags did not deploy. no_crash no_crash crash crash no_crash no_crash no_crash no_crash crash 18 sunroof is leaking. no_crash 19 motor and the frame separated from vehicle. manufacturer will be notified. 20 rear front wheel bearing broke causing vehicle to pull to the left when slowing down. consumer had brake's replaced about four times and still dealer can't determine the problem. no_crash no_crash This is the table containing the test dataset. 38 Teradata Aster R User Guide Update 3

39 doc_id text_data Chapter 4: Using Aster Analytics Functions with Aster R Using Text Analysis with Aster R 1 ELECTRICAL CONTROL MODULE IS SHORTENING OUT, CAUSING THE VEHICLE TO STALL. ENGINE WILL BECOME TOTALLY INOPERATIVE. CONSUMER HAD TO CHANGE ALTERNATOR/ BATTERY AND STARTER, AND MODULE REPLACED 4 TIMES, BUT DEFECT STILL OCCURRING CANNOT DETERMINE WHAT IS CAUSING THE PROBLEM. 2 ABS BRAKES FAIL TO OPERATE PROPERLY, AND AIR BAGS FAILED TO DEPLOY DURING A CRASH AT APPROX. 28 MPH IMPACT. MANUFACTURER NOTIFIED. 3 WHILE DRIVING AT 60 MPH GAS PEDAL GOT STUCK DUE TO THE RUBBER THAT IS AROUND THE GAS PEDAL. 4 THERE IS A KNOCKING NOISE COMING FROM THE CATALYTIC CONVERTER, AND THE VEHICLE IS STALLING. ALSO, HAS PROBLEM WITH THE STEERING. 5 CONSUMER WAS MAKING A TURN, DRIVING AT APPROX 5-10 MPH WHEN CONSUMER HIT ANOTHER VEHICLE. UPON IMPACT, DUAL AIRBAGS DID NOT DEPLOY. ALL DAMAGE WAS DONE FROM ENGINE TO TRANSMISSION, TO THE FRONT OF VEHICLE, AND THE VEHICLE CONSIDERED A TOTAL LOSS. 6 WHEEL BEARING AND HUBS CRACKED, CAUSING THE METAL TO GRIND WHEN MAKING A RIGHT TURN. ALSO WHEN APPLYING THE BRAKES, PEDAL GOES TO THE FLOOR, CAUSE UNKNOWN. WAS ADVISED BY MIDAS NOT TO DRIVE VEHICLE- WHEELE COULD COME OFF. 7 DRIVING ABOUT 5-10 MPH, THE VEHICLE HAD A LOW FRONTAL IMPACT IN WHICH THE OTHER VEHICLE HAD NO DAMAGES. UPON IMPACT, DRIVER'S AND THE PASSENGER'S AIR BAGS DID NOT DEPLOY, RESULTING IN INJURIES. PLEASE PROVIDE FURTHER INFORMATION AND VIN#. 8 THE AIR BAG WARNING LIGHT HAS COME ON, INDICATING AIRBAGS ARE INOPERATIVE. THEY WERE FIXED ONE AT THE TIME, BUT PROBLEM HAS REOCCURRED. 9 CONSUMER WAS DRIVING WEST WHEN THE OTHER CAR WAS GOING EAST. THE OTHER CAR TURNED IN FRONT OF CONSUMER'S VEHICLE, CONSUMER HIT OTHER VEHICLE AND STARTED TO SPIN AROUND, COULDN'T STOP, RESULTING IN A CRASH. UPON IMPACT, AIRBAGS DIDN'T DEPLOY. 10 WHILE DRIVING ABOUT 65 MPH AND THE TRANSMISSION MADE A STRANGE NOISE, AND THE LEFT FRONT AXLE LOCKED UP. THE DEALER HAS REPAIRED THE VEHICLE. This example shows the steps to build a Naïve Bayes Text Classifier model and then apply the model to the new log data. 1. Create a ta.data.frame consisting of tokens from the training dataset. This requires the use of the Aster Analytics function TextTokenizer(). tadf_nbayes_tokens <- ta.data.frame("select doc_id, lower(token) AS token, category FROM TextTokenizer (ON complaints PARTITION BY ANY TextColumn ('text_data') OutputByWord ('true') Accumulate ('doc_id', 'category'))", sourcetype = "query") Teradata Aster R User Guide Update 3 39

40 Chapter 4: Using Aster Analytics Functions with Aster R Using Text Analysis with Aster R 2. Train a new Naïve Bayes Text Classifier using the function aa.naivebayes.textclassifier.train(). nbayes_model_1 <- aa.naivebayes.textclassifier.train(tadf_nbayes_tokens, partition.column = "category", token.column = "token", doc.category.column = "category") The output of the function aa.naivebayes.textclassifier.train() is a list. 3. Use the "[[" operator to access the first item in the list, which is the model. nbayes_model <- nbayes_model_1[[1]] 4. Apply the model to the new log data from the test dataset. a) Create a ta.data.frame with the new log data. The new data is in an Aster table named "nbayes_test". tadf_nbayes_test <- ta.data.frame("select doc_id, lower(token) AS token FROM TextTokenizer (ON nbayes_test PARTITION BY ANY TextColumn ('text_data') OutputByWord ('true') Accumulate ('doc_id'))", sourcetype = "query") b) Predict the categories ('crash' or 'no_crash') on the new log data. aa.naivebayes.textclassifier.predict(object = nbayes_model, newdata = tadf_nbayes_test, input.token.column = 'token', doc.id.columns = 'doc_id', model.type = "BERNOULLI", model.token.column = 'token', model.category.column = 'category', model.prob.column = 'prob', newdata.partition.column = 'doc_id') The output is: $result doc_id prediction loglik 1 1 no_crash crash no_crash crash crash no_crash crash no_crash no_crash crash no_crash crash no_crash crash no_crash crash no_crash crash no_crash crash attr(,"class") [1] "aa.naivebayes.textclassifier.predict" Warning message: In ta.show(tadf, maxrows) : Printing rows in random order since base table/view is neither ordered nor have row_names column. 40 Teradata Aster R User Guide Update 3

41 Note: The "Warning message" in the output is an explanatory note to the user. Chapter 4: Using Aster Analytics Functions with Aster R Using cfilter and R Graphics with Aster R Using cfilter and R Graphics with Aster R This example uses the Aster cfilter function to examine a dataset of grocery store transactions to identify items that are often bought together. This example also shows how R graphic functions can be used with the output of Aster R functions. The input data is shown here. trans_id date store_id region item sku category west milk 1 dairy west butter 2 dairy west eggs 3 dairy west flour 4 baking west milk 1 dairy west butter 2 dairy west eggs 3 dairy west milk 1 dairy west eggs 3 dairy west flour 4 baking west milk 1 dairy west butter 2 dairy west butter 2 dairy west eggs 3 dairy west flour 4 baking west milk 1 dairy west eggs 3 dairy west eggs 3 dairy west flour 4 baking west butter 2 dairy west eggs 3 dairy west flour 4 baking 1. Convert the data to a virtual data frame. ta.droptable("shopping_tbl", schemaname = "public") Teradata Aster R User Guide Update 3 41

42 Chapter 4: Using Aster Analytics Functions with Aster R Using cfilter and R Graphics with Aster R shopping.tadf <- ta.create(shopping, table = "shopping_tbl", schemaname = "public", tabletype = "fact", partitionkey="region", row.names=true, coltypes = c( trans_id='int', date='date', store_id='int',region='text', item='text', sku='int', category='text') ) 2. Call the cfilter function. cf_out <- aa.cfilter( shopping.tadf, input.columns = "item", join.columns = "trans_id", add.columns = "region" ) The output is shown here. 3. Take the results of interest from the function output, and use the R library circlize to display these results graphically. output_table <- as.data.frame(cf_out$output.table) library(circlize) chorddiagramfromdataframe(output_table[,c("col1_item1","col1_item2","score") ]) The resulting diagram is shown here. 42 Teradata Aster R User Guide Update 3

43 Chapter 4: Using Aster Analytics Functions with Aster R Using VARMAX with Aster R Using VARMAX with Aster R This section uses the dataset "lap", found in the R package "astsa". The dataset consists of 508 mean weekly observations of eleven variables including three measurements of human death rates (total mortality, cardiac-related mortality, and respiratory-related mortality) and eight atmospheric measurements (temperature, relative humidity, carbon monoxide, sulphur dioxide, nitrogen dioxide, hydrocarbons, ozone, and particulates). This example uses the VARMAX function to predict three of the time series in the dataset: cardiac mortality ('cmort'), temperature ('tempr') and particulates ('part'). Users run the VARMAX function on a shortened version of the dataset the first 456 points, and then plot the predicted values versus the actual values for the final fifty-two weeks for each of the three times series. 1. Convert the dataset from a time series (ts) object to an R data frame. library(astsa) > class(lap) [1] "mts" "ts" > lap_df<-as.data.frame(lap) > class(lap_df) [1] "data.frame" 2. Add 'seqid' and 'id' columns to prepare the data frame for use with the VARMAX function. seqid <- rep(1, 508) id <- seq(1:508) lap_df_indexed <- cbind(seqid, id, lap_df) 3. Create a table in the Aster Database to hold the data. ta.droptable( lap_data ) Teradata Aster R User Guide Update 3 43