Aster Development Environment. User Guide

Transcription

1 Aster Development Environment User Guide Release Number 6.00 Product ID: B K September 2014

2 The product or products described in this book are licensed products of Teradata Corporation or its affiliates. Teradata, Active Data Warehousing, Active Enterprise Intelligence, Applications-Within, Aprimo Marketing Studio, Aster, BYNET, Claraview, DecisionCast, Gridscale, MyCommerce, QueryGrid, SQL-MapReduce, Teradata Decision Experts, "Teradata Labs" logo, Teradata ServiceConnect, Teradata Source Experts, WebAnalyst, and Xkoto are trademarks or registered trademarks of Teradata Corporation or its affiliates in the United States and other countries. Adaptec and SCSISelect are trademarks or registered trademarks of Adaptec, Inc. AMD Opteron and Opteron are trademarks of Advanced Micro Devices, Inc. Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. Apple, Mac, and OS X all are registered trademarks of Apple Inc. Axeda is a registered trademark of Axeda Corporation. Axeda Agents, Axeda Applications, Axeda Policy Manager, Axeda Enterprise, Axeda Access, Axeda Software Management, Axeda Service, Axeda ServiceLink, and Firewall-Friendly are trademarks and Maximum Results and Maximum Support are servicemarks of Axeda Corporation. Data Domain, EMC, PowerPath, SRDF, and Symmetrix are registered trademarks of EMC Corporation. GoldenGate is a trademark of Oracle. Hewlett-Packard and HP are registered trademarks of Hewlett-Packard Company. Hortonworks, the Hortonworks logo and other Hortonworks trademarks are trademarks of Hortonworks Inc. in the United States and other countries. Intel, Pentium, and XEON are registered trademarks of Intel Corporation. IBM, CICS, RACF, Tivoli, and z/os are registered trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. LSI is a registered trademark of LSI Corporation. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are registered trademarks of Microsoft Corporation in the United States and other countries. NetVault is a trademark or registered trademark of Dell Inc. in the United States and/or other countries. Novell and SUSE are registered trademarks of Novell, Inc., in the United States and other countries. Oracle, Java, and Solaris are registered trademarks of Oracle and/or its affiliates. QLogic and SANbox are trademarks or registered trademarks of QLogic Corporation. Quantum and the Quantum logo are trademarks of Quantum Corporation, registered in the U.S.A. and other countries. Red Hat is a trademark of Red Hat, Inc., registered in the U.S. and other countries. Used under license. SAP is the trademark or registered trademark of SAP AG in Germany and in several other countries. SAS and SAS/C are trademarks or registered trademarks of SAS Institute Inc. SPARC is a registered trademark of SPARC International, Inc. Symantec, NetBackup, and VERITAS are trademarks or registered trademarks of Symantec Corporation or its affiliates in the United States and other countries. Unicode is a registered trademark of Unicode, Inc. in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and company names mentioned herein may be the trademarks of their respective owners. THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED ON AN "AS-IS" BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO THE ABOVE EXCLUSION MAY NOT APPLY TO YOU. IN NO EVENT WILL TERADATA CORPORATION BE LIABLE FOR ANY INDIRECT, DIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS OR LOST SAVINGS, EVEN IF EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. The information contained in this document may contain references or cross-references to features, functions, products, or services that are not announced or available in your country. Such references do not imply that Teradata Corporation intends to announce such features, functions, products, or services in your country. Please consult your local Teradata Corporation representative for those features, functions, products, or services available in your country. Information contained in this document may contain technical inaccuracies or typographical errors. Information may be changed or updated without notice. Teradata Corporation may also make improvements or changes in the products or services described in this information at any time without notice. To maintain the quality of our products and services, we would like your comments on the accuracy, clarity, organization, and value of this document. Please teradata-books@lists.teradata.com. Any comments or materials (collectively referred to as "Feedback") sent to Teradata Corporation will be deemed non-confidential. Teradata Corporation will have no obligation of any kind with respect to Feedback and will be free to use, reproduce, disclose, exhibit, display, transform, create derivative works of, and distribute the Feedback and derivative works thereof without limitation on a royalty-free basis. Further, Teradata Corporation will be free to use any ideas, concepts, know-how, or techniques contained in such Feedback for any purpose whatsoever, including developing, manufacturing, or marketing products or services incorporating Feedback.

3 Copyright by Teradata. All Rights Reserved. Aster Development Environment User Guide 3

4 4 Aster Development Environment User Guide

5 Table of Contents Chapter 1: Introduction Chapter 2: Installing ADE Installation Prerequisites Verify the Java Development Kit Version Verify the Eclipse IDE Version Add Text File Associations to Support UTF Installing Eclipse Bundled with the ADE Plug-in Installing the ADE Eclipse Plug-in Updating ADE Removing ADE Troubleshooting Linux ADE Installation Issues Chapter 3: Using ADE ADE Projects Creating an Aster Database Project Configuring a Database Connection Adding Third-Party JARs Creating a SQL-MapReduce Function Creating the SQL-MapReduce Function Stub Code Defining the SQL-MR Function Creating an Input Schema Creating Output Columns Adding SQL-MapReduce Function Implementation Creating a SQL-Graph Function Creating the SQL-Graph Function Stub Code Defining the SQL-GR Function Creating an Input Schema Creating Output Columns Creating an Aggregator Function Creating the Aggregator Function Stub Code Defining the Aggregator Function Aster Development Environment User Guide 5

6 Creating an Input Schema Creating Output Columns Testing SQL-MapReduce Functions Locally Creating a Test Creating Input Data Running the Test Creating Queries to Invoke SQL-MapReduce Functions Creating a Sample Table Creating a Query Compiling, Deploying, and Running Functions Managing Functions and Files on Aster Database Clusters Exporting SQL-MapReduce Functions Making SQL-MapReduce Functions Available to Any Aster Database Cluster Viewing Aster Database Cluster Information Uploading Functions and Files to Aster Database Downloading, Removing, or Moving Files on Aster Database Downloading a File or Function Removing a File or Function Moving a File or Function From One Cluster to Another Managing Database Connections Index Aster Development Environment User Guide

7 CHAPTER 1 Introduction Aster Development Environment (ADE) is a development environment that lets you write, test, and deploy your SQL-MapReduce, SQL-Graph and Aggregator functions on Aster Database. Teradata provides ADE in either of two packages: As a plug-in that you can use with the Eclipse IDE for Java Developers. As a full version of Eclipse with the plug-in included. This document describes how to install and use ADE and guides you through the process of creating and deploying SQL-MapReduce functions, and creating SQL-Graph and Aggregator functions. For more information about SQL-MapReduce functions, see the Aster Developer Guide. For more information about SQL-GR (Graph) and Aggregator functions, see the Aster Developer Guide. Aster Development Environment User Guide 7

8 Introduction 8 Aster Development Environment User Guide

9 CHAPTER 2 Installing ADE Installation Prerequisites Installing Eclipse Bundled with the ADE Plug-in Installing the ADE Eclipse Plug-in Updating ADE Removing ADE Troubleshooting Linux ADE Installation Issues Tip: The Eclipse procedures in this section are for the Windows operating system. Both Eclipse and Aster Development Environment run on Windows, MAC OS X, and Linux, although the specific steps for each operating system are not listed here. Installation Prerequisites Before installing the ADE the following Eclipse prerequisites must be met. Verify the Java Development Kit Version Before installing the ADE, install or verify that the version of your Java Development Kit (JDK) is 32-bit, and not 64-bit. Ensure it is installed and running on your Windows XP, Windows 7, MAC OS X, or Linux system. Verify the Eclipse IDE Version This section only applies if you already have Eclipse installed, and are adding the ADE plug-in for Eclipse. If you are downloading the full version of Eclipse with the plug-in included from Teradata, you can skip this requirement. Before installing the ADE plug-in, install or verify the correct version number of the Eclipse IDE for Java Developers. Ensure that version or higher of Eclipse IDE for Java Developers is installed and running on your Windows XP, Windows 7, MAC OS X, or Linux system. Aster Development Environment User Guide 9

10 Installing ADE Installation Prerequisites Add Text File Associations to Support UTF-8 If your Java text files contain Unicode characters for languages such as Chinese or Japanese, you must configure support for UTF-8. Once UTF-8 encoding is set, the Java files can be saved as UTF-8 in the ADE and the Unicode characters will display correctly in both the window console and the corresponding output file. In order to support the UTF-8 character set, two new text file associations must be added and configured in Eclipse: *.in (for the default input file) *.out(for the default output file) These files allow a developer to create a TestRunner test file in Eclipse in order to test code locally as you work on it, without an Aster Database connection. See Testing SQL- MapReduce Functions Locally on page 35 for more information. To add text file associations to support UTF-8: 1 Launch Eclipse and navigate to Window > Preferences. 2 In the Preferences window navigate to General > Content Types > Text > and click the Add button in the File associations section. 10 Aster Development Environment User Guide

11 Installing ADE Installing Eclipse Bundled with the ADE Plug-in 3 In the Add Content Type Association dialog box enter *.in and click OK. 4 In the Default encoding dialog box enter UTF-8 and click Update. 5 Repeat the previous two steps to add *.out content type and its UTF-8 default encoding. The File associations section of this window displays the two added files and they may be edited or removed as necessary. This screen also displays three additional content types (*.sql, *.test, and *.txt) that are added by default. Theses default types are locked and cannot be removed or edited. Installing Eclipse Bundled with the ADE Plug-in You can install the full version of Eclipse with the ADE plug-in included, if you do not already have Eclipse installed. The installer name depends upon your operating system: For Windows: ADE-win32.win32.x86.zip For Linux 64-bit: ADE-linux.gtk.x86_64.tar.gz For Linux 32-bit: ADE-linux.gtk.x86.tar.gz For MacOS: ADE-macosx.cocoa.x86.tar.gz Extract the file to the directory where you want to install Eclipse. Installing the ADE Eclipse Plug-in The process described here is for adding the ADE plug-in to an existing Eclipse installation. The ADE plug-in runs on Windows XP, Windows 7, MAC OS X, and Linux. To install the ADE plug-in: 1 Obtain the Aster Database ADE plugin package (ADE-plugin-only.zip). 2 Extract the contents of ADE-plugin-only.zip to a folder on your system. For example, if you extract the contents of the ZIP file to C:\ on Windows, the extraction tool creates the C:\ADE-plugin-only folder and stores the ADE installer in it. 3 (For Linux Only) Set the GDK_NATIVE_WINDOWS environment variable. export GDK_NATIVE_WINDOWS=1 4 Launch Eclipse and choose Help > Install New Software. 5 In the Install window, click Add. 6 In the Add Site dialog box, click Local. 7 In the Browse for Folder dialog box, select the ADE-plugin-only folder you created in Step 2 and click OK. Aster Development Environment User Guide 11

12 Installing ADE Installing the ADE Eclipse Plug-in 8 (Optional) In the Name field, enter a name of the site where the plugin is added and click OK. An Aster Data entry appears in the list of available software. If not, from the Work with drop-down menu, choose --All Available Sites-- or the name of the site you created in this step. The Aster Data entry appears in the list. 9 Expand the Aster Data node and check the Aster Development Environment check box and click Next. 10 In the Install Details screen, select Aster Development Environment from the list. 11 Click Finish to start the plugin installation process. 12 When prompted to continue the installation process, click OK. This message appears because the ADE plugin contains unsigned content. 12 Aster Development Environment User Guide

13 Installing ADE Installing the ADE Eclipse Plug-in 13 If the Do you trust these certificates? dialog box appears, check the check box to accept the Eclipse.org Foundation certificate and click OK. 14 Click Yes to restart Eclipse. 15 After Eclipse restarts, choose Other from the Perspective Chooser drop-down menu. 16 In the Open Perspective window, choose Aster SQL-MapReduce and click OK to activate the Aster Development Perspective and finish the ADE installation. Aster Development Environment User Guide 13

14 Installing ADE Updating ADE 17 To confirm that ADE is installed, open the File > New menu. The following items are visible in Eclipse: Aster SQL-MapReduce Project Aster SQL-MapReduce Function Aster SQL-Graph Function Aster Aggregator Function Aster SQL File Aster SQL-MapReduce Test Configuration Go to Using ADE on page 17 for instructions on how to create a SQL-MapReduce function. Updating ADE To update ADE: 1 Launch Eclipse and Choose Help > Install New Software. 2 In the Install window, click the What is already installed? link. 14 Aster Development Environment User Guide

15 Installing ADE Removing ADE 3 In the Installation Details window, click Aster Development Environment. 4 Click Update. Removing ADE To remove ADE from Eclipse: 1 Launch Eclipse. 2 Choose Window > Close All Perspectives. You cannot remove ADE if the Aster Development perspective is not closed. 3 Choose Help > Install New Software. 4 In the Install window, click the What is already installed?. 5 In the Installation Details window, click Aster Development Environment. 6 Click Uninstall. Aster Development Environment User Guide 15

16 Installing ADE Troubleshooting Linux ADE Installation Issues 7 In the Uninstall window, click Finish. Troubleshooting Linux ADE Installation Issues If the GDK_NATIVE_WINDOWS environment variable was not set in Step 3 in the Installing ADE section, there may be buttons in ADE that do not show up or do not respond when clicked or selected. This known issue affects the ADE on Ubuntu To resolve this issue: a Set the GDK_NATIVE_WINDOWS variable in the command-line: export GDK_NATIVE_WINDOWS=1 b Run Eclipse or run ADE directly. To run ADE directly, use this CLI command:./asterdevelopmentenvironment If errors occur when launching ADE, check GTK permissions on Ubuntu. To resolve this issue, run this command: gksudo./asterdevelopmentenvironment 16 Aster Development Environment User Guide

17 CHAPTER 3 Using ADE ADE Projects on page 17 Creating an Aster Database Project on page 18 Creating a SQL-MapReduce Function on page 22 Creating a SQL-Graph Function on page 29 Creating an Aggregator Function on page 32 Testing SQL-MapReduce Functions Locally on page 35 Creating Queries to Invoke SQL-MapReduce Functions on page 41 Compiling, Deploying, and Running Functions on page 43 Managing Functions and Files on Aster Database Clusters on page 44 Managing Database Connections on page 49 ADE Projects An ADE project serves two purposes: It is the container that holds your: SQL-MapReduce code, related libraries, and tests SQL-Graph code Aggregator code It provides the connection detail that allows your code to connect to Aster Database. Aster Development Environment User Guide 17

18 Creating an Aster Database Project Creating an Aster Database Project To create an ADE project: 1 Launch Eclipse. 2 Choose File > New > Project. The New Project wizard window appears. 3 In the Select a wizard screen, select Aster Analytics Project and click Next. 4 In the Create an Aster Analytics Project screen: a In the Project name field, enter the name of the project (only use letters, numerals, and hyphens). 18 Aster Development Environment User Guide

19 Creating an Aster Database Project b c d e If you have not configured an Aster Database connection, or if you need to configure a new one, click the Configure Aster Database link and follow the steps in Configuring a Database Connection on page 20 before proceeding to the next step. To set this project to always use a particular database connection, click the Use a specific Aster Database radio button and choose the database from the drop-down menu and click Next. Alternatively, to use the currently selected database connection in your Eclipse project, click the Use a workspace-wide Active Aster Database radio button and click Next. In the Java Settings screen, click Finish. f If prompted to confirm whether to open the Aster SQL-MapReduce perspective, click Yes. Aster Development Environment User Guide 19

20 Creating an Aster Database Project The new project appears in the Package panel. g To display that the Aster Database SQL-MapReduce libraries, expand the project. 5 (Optional) To add third-party project libraries or other prebuilt code to your project, follow the steps in Adding Third-Party JARs on page 21. You are now ready to create a SQL-MapReduce function. Configuring a Database Connection The Preferences (Filtered) window is where the new Aster Database connection is added and configured. The Configure Aster Database link in the Create an Aster SQL-MapReduce Project screen, lets you add and configure new Aster Database connection. To add and configure a new Aster Database connection: 1 In the Preferences (Filtered) window, click Add. 20 Aster Development Environment User Guide

21 Creating an Aster Database Project 2 In the Add New Aster Database Connection window, complete the following fields: Name: The name of the database connection. Hostname: The hostname or IP address of the Aster Database queen that hosts that database. Database: The database name. Username and Password: The username and password for accessing the database. 3 To ensure that the ADE project can connect to the database, click Check connection. 4 To add the connection, click OK. Adding Third-Party JARs By default, the connection that you add becomes the active connection. If not, or to make another connection active, select it and click Set as Active. 5 To close the Preferences (Filtered) window click OK. To add third-party libraries or other prebuilt code to your project: 1 Right-click the name of your project in the Package panel, and choose Build Path > Add External Archives. 2 Select the jars you want to include and click Open. The jar appears in your project under Referenced Libraries. 3 Right-click your project and choose Build Path > Configure Build Path. 4 In the Properties window, click Java Build Path in the navigation tree. Aster Development Environment User Guide 21

22 Creating a SQL-MapReduce Function 5 In the Order and Export panel, click Up/Down to change the order of the library in the list. 6 Select the library by clicking its check box. 7 Click OK. Creating a SQL-MapReduce Function To create a SQL-MapReduce (SQL-MR) function, perform these steps: Creating the SQL-MapReduce Function Stub Code on page 22 Adding SQL-MapReduce Function Implementation on page 26 Creating the SQL-MapReduce Function Stub Code To create the SQL-MapReduce function stub code, perform these steps: Defining the SQL-MR Function on page 22 Creating an Input Schema on page 24 Creating Output Columns on page 25 Defining the SQL-MR Function To define a SQL-MR function: 1 In Eclipse s Package panel, select your ADE project. 2 Choose File > New > Aster SQL-MapReduce Function. 22 Aster Development Environment User Guide

23 Creating a SQL-MapReduce Function The New Aster Data Function wizard appears. 3 In the SQL-MapReduce Function screen: a b c d (Optional) in the Package field, enter the name of the package or click Browse, select a package and click OK. In the Name field, enter the name of the function. Select the interfaces that this function implements by checking the check boxes that apply. In this example, select Single Input RowFunction. For information about row, partition, and multiple input functions, see the Aster Developer Guide. The use of the default package is discouraged when Single Input is selected. Click Next. 4 In the Argument clauses screen, set up the function to accept arguments from the query that calls the function. These are the parameter settings that you provide so that the user can specify how the function should operate in the context of the user s query. To add an argument clause: a To an argument clause row to the list, click Add. b Define the argument clause: In Name cell, enter the name of the column. In the Type cell, choose the type of the argument from the drop-down menu. In the Optional cell, check the check box if the argument clause is optional. In the List cell, check the check box to provide a list of comma-separated acceptable values for the argument, and enter them in the adjacent cell. Aster Development Environment User Guide 23

24 Creating a SQL-MapReduce Function c Click Next. An introduction screen appears describing the either single- or multiple-input functions. In this example, Introduction Page screen describes what a single-input function is. 5 In the Introduction Page screen click Next. 6 In the Input Schema screen, create an input schema, as described in Creating an Input Schema. Creating an Input Schema Depending on your interface selection, you may need to create either a single-input schema or a multiple-input one. Creating a Single-Input Function Schema on page 24 Proceed to Creating Output Columns on page 25.Creating a Multiple-Input Function on page 25 Creating a Single-Input Function Schema To create a single-input function schema: 1 In the Input Schema for single-input function screen, specify the types of columns your single input row function takes as input. Do one of the following: To accept all columns, click Accept all input columns (you can add restrictions in your SQL-MapReduce Java code later). To specify certain columns only, choose Accept columns with exactly these types, click Add, and choose a column type from the drop-down menu. In this example, choose integer. To ensure that the your chosen column types match those of a particular table or view that you plan to use as input to the function, click Query Aster Database. This button connects to the database and opens a window that lets you choose the table column whose type you want to add to the list. 24 Aster Development Environment User Guide

25 Creating a SQL-MapReduce Function 2 Click Next. 3 Proceed to Creating Output Columns on page 25.Creating a Multiple-Input Function To create a multiple-input function schema: 1 In the Input Schema for multiple-input function window, specify the number of inputs your function accepts: To specify any number of inputs, click Any number of Inputs. To specify any number of inputs, click A range of Inputs, and enter the minimum and maximum (optional) number of inputs. To specify an exact number of inputs, click Exactly Inputs, and enter the number of inputs. Also, click Add, and specify the input name and input kind your function accepts. There are rules that govern the number and kinds of input allowed. See the Aster Developer Guide for more information on these rules and the different input kinds. 2 Click Next. 3 Proceed to Creating Output Columns on page 25. Creating Output Columns To define the output column names and types: Aster Development Environment User Guide 25

26 Creating a SQL-MapReduce Function 1 In the Output Columns screen, choose the output column names and types: To return all the input columns, choose Exactly the input column names and types. To return all the input columns in addition to the columns listed in the list, choose The input columns with the below columns added. Click Add to add the additional columns. To return only the columns specified in the list, choose Exactly these names and types. If you choose this option, the output columns need not match the names or types of the input columns. Click Add to add the output columns. In this example, choose Exactly these names and types, click Add, set the Name cell to cuberootresult, and choose numeric from the Type drop-down menu. 2 Click Finish. ADE generates stub code for the function. 3 In Eclipse, choose File > Save. Adding SQL-MapReduce Function Implementation In the Java stub code that ADE generates, the constructor specifies mapping between input and output columns of the function. 26 Aster Development Environment User Guide

27 Creating a SQL-MapReduce Function In most cases you can use the constructor as-is, however, you need to add your implementation of the SQL-MapReduce function to one of these important methods: operateonsomerows() The row SQL-MapReduce function. operateonpartition() The partition SQL-MapReduce function. operateonmultipleinputs() The multiple-input SQL-MapReduce function. In this example, you implement a row function. To add your implementation to one of these functions: 1 (Optional) Before adding your implementation, create a TestRunner test file in Eclipse that lets you test your code locally (no Aster Database needed) as you work on it. For instructions on how to create the test file, see Testing SQL-MapReduce Functions Locally on page 35. Aster Development Environment User Guide 27

28 Creating a SQL-MapReduce Function 2 In Eclipse, open the Java class generated by ADE. In this example, the class is called CubeRoot.java. 3 In the SQL-MapReduce function, in this example operateonsomerows(), add your implementation to the while(inputiterator.advancetonextrow()) loop: a Delete the exception code: throw new ClientVisibleException( operateonsomerows() needs to be implemented! ); These lines serve as a hint that the implementation has not yet started. b In the place of deleted exception code, provide your SQL-MapReduce function implementation. For example, this code implements a cube-root row function: public void operateonsomerows(rowiterator inputiterator,rowemitter outputemitter) { // // This method will be called once for each set of rows. // 28 Aster Development Environment User Guide

29 Creating a SQL-Graph Function while (inputiterator.advancetonextrow()) { // // Construct output rows here using calls like // outputemitter.addint(), // outputemitter.addstring(), or outputemitter.addfromrow(). // Once finished building the row, call outputemitter.emitrow(). // int value = inputiterator.getintat(0); double root = Math.pow(value, 1.0/3); outputemitter.adddouble(root); outputemitter.emitrow(); } } 4 To test your code locally, go to Testing SQL-MapReduce Functions Locally on page To create a SQL query that invokes your function, go to Creating Queries to Invoke SQL- MapReduce Functions on page 41. For more information about implementing row and partition functions, see the Aster Developer Guide. Creating a SQL-Graph Function This section describes how to create a SQL-Graph function. For more information on implementing SQL-Graph functions, see the Aster Developer Guide. Creating the SQL-Graph Function Stub Code To create the SQL-Graph function stub code, perform these steps: Defining the SQL-GR Function on page 29 Creating an Input Schema on page 31 Creating Output Columns on page 31 Defining the SQL-GR Function To define a SQL-GR function: 1 In Eclipse s Package panel, select your ADE project. 2 Choose File > New > Aster SQL-Graph Function. The New Aster Data Function wizard appears. a b On the Introduction Page to Graph function screen, click Next. In the Name field on the SQL-GRAPH Function screen, enter the name of the function. Click Next. Aster Development Environment User Guide 29

30 Creating a SQL-Graph Function c In the Argument clauses screen, set up the function to accept arguments from the query that calls the function. These are the parameter settings that you provide so that the user can specify how the function should operate in the context of the user s query. To add an argument clause row to the list: i Click Add. ii iii Define the argument clause: In Name cell, enter the name of the column. In the Type cell, choose the type of the argument from the drop-down menu. In the Optional cell, check the check box if the argument clause is optional. In the List cell, check the check box to provide a list of comma-separated acceptable values for the argument, and enter them in the adjacent cell. Click Next. 3 Proceed to Creating an Input Schema on page Aster Development Environment User Guide

31 Creating a SQL-Graph Function Creating an Input Schema To create an input schema: 1 In the Input Schema for SQL-GRAPH function screen, specify the input accepted by the graph function. a b c Under the Require inputs with the following Alias Names and Types heading: Click Add. Specify the Input Name and select the InputKind accepted by the function. There are rules that govern the number and kinds of input allowed. See the Aster Developer Guide for more information on these rules and the different input kinds. Under the Which input table will be used to obtain VertexKey schema heading: Click Add. In the Select column screen, select the column for the VertexKey schema and click OK. Under the Vertex Message Schema Definition heading: Click Add. Select the type of vertex message schema definition. Tip: If you check the Enable distinct Vertex Message check box, SQL-GR sends only distinct messages to vertices. For example, if the message consists of one integer value, if multiple messages have the same value, only one message is passed; the remaining messages are discarded. d Click Next. 2 Proceed to Creating Output Columns on page 31. Creating Output Columns To define the output column names and types: Aster Development Environment User Guide 31

32 Creating an Aggregator Function 1 In the Output Columns screen, click Add to specify the output column names and types. The output columns do not need to match the names or types of the input columns 2 Click Finish. ADE generates stub code for the function. 3 In Eclipse, choose File > Save As... For more information on implementing SQL-Graph functions, see the Aster Developer Guide. Creating an Aggregator Function This section describes how to create an Aggregator function. For more information on implementing Aggregator functions, see the Aster Developer Guide. Creating the Aggregator Function Stub Code To create the Aggregator function stub code, perform these steps: Defining the Aggregator Function on page 32 Creating an Input Schema on page 34 Creating Output Columns on page 35 Defining the Aggregator Function To define an Aggregator function: 1 In Eclipse s Package panel, select your ADE project. 2 Choose File > New > Aster Aggregator Function. The New Aster Data Function wizard appears. a On the Introduction Page to Graph function screen, click Next. 32 Aster Development Environment User Guide

33 Creating an Aggregator Function b In the Name field on the Aggregator Function screen, enter the name of the function. Click Next. c In the Argument clauses screen, set up the function to accept arguments from the query that calls the function. These are the parameter settings that you provide so that the user can specify how the function should operate in the context of the user s query. To add an argument clause row to the list: i Click Add. ii iii Define the argument clause: In Name cell, enter the name of the column. In the Type cell, choose the type of the argument from the drop-down menu. In the Optional cell, check the check box if the argument clause is optional. In the List cell, check the check box to provide a list of comma-separated acceptable values for the argument, and enter them in the adjacent cell. Click Next. d Proceed to Creating an Input Schema on page 34. Aster Development Environment User Guide 33

34 Creating an Aggregator Function Creating an Input Schema To create an input schema: 1 In the Input Schema for Aggregator function screen, specify the input column names and types accepted by the aggregator function. a b Click Add. Specify the input column Name and select the Type accepted by the function. c Click Next. 2 In the Partial Input Schema for Aggregator function screen, specify the types of partial schema accepted by the aggregator function. a b Click Add. Specify the type of partial schema accepted by the function. c Click Next. 3 Proceed to Creating Output Columns on page Aster Development Environment User Guide

35 Testing SQL-MapReduce Functions Locally Creating Output Columns To define the output column types: 1 In the Output Columns screen, click Add to specify the output column types. The output columns do not need to match the names of the input columns 2 Click Finish. ADE generates stub code for the function. 3 In Eclipse, choose File > Save As... For more information on implementing Aggregator functions, see the Aster Developer Guide Testing SQL-MapReduce Functions Locally Creating a Test ADE includes a feature called TestRunner that lets you create a test file in Eclipse to test your code locally, without an Aster Database connection. To test your SQL-MapReduce function, perform these steps: Creating a Test on page 35 Creating Input Data on page 38 Running the Test on page 38 To create your test in TestRunner: Aster Development Environment User Guide 35

36 Testing SQL-MapReduce Functions Locally 1 Choose File > New > Aster SQL-MapReduce Test Configuration. The New Test Configuration wizard appears. 2 In the Aster SQL-MapReduce Test Configuration screen of the wizard, configure the test settings. For this example, keep the default values. This is a description of the fields in this screen: Folder The folder where the test file is saved. Filename The name of the test file containing the test data and settings. Function name The name of the Java class that implements your SQL-MapReduce function (in this example, CubeRoot). Which type of SQL-MR function will the SQL invoke? The type of SQL-MR function you are testing, either Single Input Function or Multiple Input Function. Argument Classes If your function takes arguments, click Add and add a name/value pair for each argument your test passes to your SQL-MR function. 3 Click Next. 36 Aster Development Environment User Guide

37 Testing SQL-MapReduce Functions Locally 4 In the Input screen, define the source and schema of the input data the test uses. a b c d e Click Use File. Click comma-separated values (CSV). To add an input, click Add and specify the input column name and type. In this example, add an integer input. Enter or browse to the Input File to specify the input file to be used in the test. NOTE: If an input file *.in exists before running the TestRunner configuration UI wizard, the ADE will not generate a new input file. Instead, the ADE will use the existing *.in input file. Rename your files, if necessary, before running the TestRunner. Click Next. 5 In the Define Partitioning and Ordering screen, define how the input data should be partitioned and reordered. Then, click Next. For this example, no need to make any changes to this screen. 6 In the Output screen, specify the format and destination of the test output. You can direct the output to a file or to the Eclipse console. For this example, send the results to the console by clicking Use the console. 7 Click Finish. ADE generates the following: Test file (its name ends with.test) If specified, the output data file (its name ends with.out) If specified, the input data file (its name ends with.in) Aster Development Environment User Guide 37

38 Testing SQL-MapReduce Functions Locally These files appear in the Eclipse Package panel. 8 Proceed to Creating Input Data on page 38. Creating Input Data If your test uses an input data file, edit it in Eclipse: 1 In the Package panel, select the input data file. For this example, select CubeRoot.in. 2 To open the file for editing, double-click it. 3 Enter the test input values. 4 Click Save. 5 Proceed to Running the Test on page 38. Running the Test To test your SQL-MapReduce function, run the test you created in Eclipse: 38 Aster Development Environment User Guide

39 Testing SQL-MapReduce Functions Locally 1 Double-click the test in the Package panel. In this example, double-click CubeRoot.test. 2 Choose Run > Run As > Locally in TestRunner. 3 If prompted to save sources, click OK. The results of the test appear in the Console panel. 4 When testing your function locally with a.test file, you can use the debugging tools in Eclipse. Aster Development Environment User Guide 39

40 Testing SQL-MapReduce Functions Locally For example: a Right-click in the margin of the Java class that implements your SQL-MR function and choose Toggle Breakpoint to set a breakpoint. b Debug the test file by choosing the test file from the Debug drop-down menu in the tool bar. Running with breakpoints launches the Debug perspective in Eclipse, where you can step through your code to pinpoint errors. 40 Aster Development Environment User Guide

41 Creating Queries to Invoke SQL-MapReduce Functions Creating Queries to Invoke SQL-MapReduce Functions Creating a Sample Table You can invoke SQL-MapReduce functions in SQL SELECT queries. ADE helps you write the basic query that invokes your function. This section describes how you can create a sample query that invokes a sample SQL-MapReduce function. Creating a Sample Table on page 41 Creating a Query on page 42 For more information about invoking SQL-MapReduce functions in SQL SELECT queries, see the Aster Developer Guide. For testing purposes, it is convenient to create a table with sample input data for the query. To create a sample table in Eclipse: 1 Select Aster SQL-MapReduce perspective. 2 Choose File > New > Aster SQL File. 3 Click Browse and select the folder where the query is stored. 4 In the New SQL File window, in the File name field, enter example_table.sql. 5 Click Finish. Aster Development Environment User Guide 41

42 Creating Queries to Invoke SQL-MapReduce Functions The new SQL file appears in the Package panel. Creating a Query 6 Delete the provided code and replace it with this code, which creates a table and adds data to it: -- Set up example table CREATE FACT TABLE to_root ( INPUT int, PARTITION KEY (input) ); -- Place test input data in table INSERT INTO to_root VALUES (10000), (999), (12), (3), (2432), (1000), (8); 7 Click Save. To create a query in Eclipse: 1 Select File > New > Aster SQL File to generate the stub SQL query. 2 Click Browse and select the folder where the query is stored. 3 In the File name field, enter get_cube_root.sql. 4 Click Finish. The get_cube_root.sql file appears in the Package panel. 5 Edit the query to match your function: a b c d e f In the SELECT clause, specify the table columns to be retrieved, or * for all columns. Replace functionname with a name for your function. Replace the table-or-query for each ON clause with the name of your input table. If this function contains partitioned inputs, edit the PARTITION BY clause to partition the input data. If needed, edit the ORDER BY clause to sort the rows within the partitions. If this function is a multiple-input function that contains an input that uses data as it is already stored, use PARTITION BY ANY. If this function is a multiple -input function that contains dimensional inputs, use the DIMENSION clause to replicate the input data. 42 Aster Development Environment User Guide

43 Compiling, Deploying, and Running Functions g If your SQL-MapReduce function takes arguments, replace clausename (arg) with the name of your first name/value argument pair, and add more argument pairs as needed, separated by commas. 6 For this example, replace the code provided by ADE with the following: -- Example query using cuberoot function SELECT * FROM cuberoot( ON to_root ); 7 Click Save. 8 Go to Compiling, Deploying, and Running Functions on page 43. Compiling, Deploying, and Running Functions After writing an SQL-MR function and an SQL query to invoke the function, compile and deploy the files on Aster Database. To compile and deploy the files on Aster Database: 1 In the Eclipse, in the Package panel, select the SQL file that creates the sample input table. 2 Right-click the file name and choose Run As > Target Aster Database. 3 If prompted to save resources, click OK. ADE automatically builds and deploys the SQL-MapReduce functions in the current project, then executes the SQL file on Aster Database. The output appears in the Console panel. 4 In the Package panel, select the SQL query file that calls your SQL-MapReduce function. 5 Right-click the file and choose Run As: Target Aster Database. ADE automatically builds and deploys the SQL-MapReduce functions in the current project, then executes the SQL file on Aster Database. The output appears in the Console panel. Aster Development Environment User Guide 43

44 Managing Functions and Files on Aster Database Clusters If the query fails, edit the SQL or your Java code in Eclipse, save it, and invoke the Run As command again. Each time you do this, your SQL-MapReduce functions are uploaded to Aster Database, replacing the old versions there. After you have run the function in this manner, it remains available in the database where you installed it. Within a database, SQL-MapReduce functions reside in the public schema by default, so any user of the database can make use of it. If you also need it in other databases, you must deploy it separately to each database. Managing Functions and Files on Aster Database Clusters You can use ADE to manage the functions and files deployed on Aster Database clusters that are accessible from your ADE project. This is useful for all sorts of administrative purposes. For example, you can make executable files available for use on other clusters, see what has been installed on a given cluster, uninstall files from clusters, and move files from one cluster to another. Exporting SQL-MapReduce Functions on page 44 Viewing Aster Database Cluster Information on page 46 Uploading Functions and Files to Aster Database on page 47 Downloading, Removing, or Moving Files on Aster Database on page 48 Exporting SQL-MapReduce Functions Once you have built your SQL-MapReduce function (as described in Compiling, Deploying, and Running Functions on page 43) you can save the compiled SQL-MapReduce function as a jar that you can deploy on any Aster Database cluster. To export SQL-MapReduce Functions: 44 Aster Development Environment User Guide

45 Managing Functions and Files on Aster Database Clusters 1 In the Eclipse, in the Package panel, expand your project and Deployment nodes. Your compiled project appears under Deployment. The file ends with.zip. 2 Right-click the zip file and choose Export. 3 Select Java > JAR file and click Next. 4 Click Browse, select the export destination, specify the name of the JAR file, and click Save. 5 Click Finish. Making SQL-MapReduce Functions Available to Any Aster Database Cluster To make the JAR file containing SQL-MR functions available for uploading to any Aster Database cluster: 1 Import the file into your ADE project in Eclipse by choosing File > Import. 2 In the Select screen of the Import wizard, expand the General node and double-click File System. Click Next. Aster Development Environment User Guide 45

46 Managing Functions and Files on Aster Database Clusters 3 Click Browse for the From directory field, select the folder containing the file to import, and click OK. 4 Check the check box of the files to import. 5 Click Browse for the Into folder field, then select your project. The file appears in the Package panel. To install the JAR file on an Aster Database cluster, see Uploading Functions and Files to Aster Database. Viewing Aster Database Cluster Information To see installed files and other information about the Aster Database clusters that are configured as part of your ADE project: 46 Aster Development Environment User Guide

47 Managing Functions and Files on Aster Database Clusters 1 In Eclipse, in the Package panel, click the name of your project, then choose Project > Properties. 2 In the Properties window, expand the Aster Development Environment node, select Aster Database, then click the Configure Aster Database link. 3 In the Preferences (Filtered) window, select an Aster Database, then click Info. The Aster Database Information window appears. This window displays the files that are currently installed on the cluster. Uploading Functions and Files to Aster Database You can use ADE to install SQL-MapReduce functions and other files onto your Aster Database clusters. You can install: Aster Development Environment User Guide 47

48 Managing Functions and Files on Aster Database Clusters SQL-MapReduce functions Compiled Java and C executables that can be called by name in the FROM clause of a SELECT statement. Files Installed files are typically used to provide settings to SQL-MapReduce functions. Installed files can only be used in your SQL-MapReduce functions. Installed files cannot be directly invoked or accessed via SQL. To install a files or a function: 1 Display the Aster Database Information window as described in Viewing Aster Database Cluster Information. 2 Click Upload File. 3 In the Select file window, in the Select a resource to open field, type part of the filename or a regular expression. Matching files are displayed below. 4 Select the file you want to upload, and click OK. Downloading, Removing, or Moving Files on Aster Database Perhaps you only need to install files on one Aster Database. In that case, the procedure in the previous section, Uploading Functions and Files to Aster Database, might be all you need. However, if you have several clusters, you might need to move files around from one to another. And it is likely that you might need to uninstall a file from a cluster at some point. You can download, remove, and move files through the Aster Database Information window, as described in this section. Downloading a File or Function on page 48 Removing a File or Function on page 49 Moving a File or Function From One Cluster to Another on page 49 Downloading a File or Function To download a file or function from a cluster: 48 Aster Development Environment User Guide