Client Usage. Client Usage. Assumptions. Usage. Hdfs.put( session ).file( localfile ).to( remotefile ).now() java -jar bin/shell.
|
|
- Barnard Mervyn Long
- 6 years ago
- Views:
Transcription
1 Client Usage Client Usage Hadoop requires a client that can be used to interact remotely with the services provided by Hadoop cluster. This will also be true when using the Apache Knox Gateway to provide perimeter security and centralized access for these services. The two primary existing clients for Hadoop are the CLI (i.e. Command Line Interface, hadoop) and HUE (i.e. Hadoop User Environment). for several reasons however, neither of these clients can currently be used to access Hadoop services via the Apache Knox Gateway. This lead to thinking about a very simple client that could help people use and evaluate the gateway. The list below outline the general requirements for such a client Promote the evaluation and adoption of the Apache Knox Gateway Simple to deploy and use on data worker desktops to access to remote Hadoop clusters Simple to extend with new commands both by other Hadoop projects and by the end user Support the notion of a SSO session for multiple Hadoop interactions Support the multiple authentication and federation token capabilities of the Apache Knox Gateway Promote the use of REST APIs as the dominant remote client mechanism for Hadoop services Promote the the sense of Hadoop as a single unified product Aligned with the Apache Knox Gateway's overall goals for security The result is a very simple DSL ( Domain Specific Language) of sorts that is used via Groovy scripts. Here is an example of a command that copies a file from the local file system to HDFS. Note: The variables session, localfile and remotefile are assumed to be defined. Hdfs.put( session ).file( localfile ).to( remotefile ).now() This work is very early in development but is also very useful in its current state. We are very interested in receiving feedback about how to improve this feature and the DSL in particular. A note of thanks to REST-assured which provides a Fluent interface style DSL for testing REST services. It served as the initial inspiration for the creation of this DSL. Assumptions This document assumes a few things about your environment in order to simplify the examples The JVM is executable as simply java. The Apache Knox Gateway is installed and functional. The example commands are executed within the context of the GATEWAY_HOME current directory. The GATEWAY_HOME directory is the directory within the Apache Knox Gateway installation that contains the README file and the bin, conf and deployments directories. A few examples require the use of commands from a standard Groovy installation. These examples are optional but to try them you will need Groovy installed. Usage The DSL requires a shell to interpret the Groovy script. The shell can either be used interactively or to execute a script file. To simplify use, the distribution contains an embedded version of the Groovy shell. The shell can be run interactively. Use the command exit to exit. java -jar bin/shell.jar When running interactively it may be helpful to reduce some of the output generated by the shell console. Use the following command in the interactive shell to reduce that output.
2 This only needs to be done once as these preferences are persisted. set verbosity QUIET set show-last-result false Also when running interactively use the exit command to terminate the shell. Using ^C to exit can sometimes leaves the parent shell in a problematic state. The shell can also be used to execute a script by passing a single filename argument. java -jar bin/shell.jar samples/putfile.groovy s Once the shell can be launched the DSL can be used to interact with the gateway and Hadoop. Below is a very simple example of an interactive shell session to upload a file to HDFS. java -jar bin/shell.jar knox:000> hadoop = Hadoop.login( " "bob", "bob-password" ) knox:000> Hdfs.put( hadoop ).file( "README" ).to( "/tmp/example/readme" ).now() The knox:000> in the example above is the prompt from the embedded Groovy console. If you output doesn't look like this you may need to set the verbosity and show-last-result preferences as described above in the Usage section. If you relieve an error HTTP/ Forbidden it may be because that file already exists. Try deleting it with the following command and then try again. knox:000> Hdfs.rm(hadoop).file("/tmp/example/README").now() Without using some other tool to browse HDFS it is hard to tell that that this command did anything. Execute this to get a bit more feedback. knox:000> println "Status=" + Hdfs.put( hadoop ).file( "README" ).to( "/tmp/example/readme2" ).now().statuscode Status=201 Notice that a different filename is used for the destination. Without this an error would have resulted. Of course the DSL also provides a command to list the contents of a directory.
3 knox:000> println Hdfs.ls( hadoop ).dir( "/tmp/example" ).now().string {"FileStatuses":{"FileStatus":[{"accessTime": ,"blockSize": ,"grou p":"hdfs","length":19395,"modificationtime": ,"owner":"bob","pathsuffix":" README","permission":"644","replication":1,"type":"FILE",{"accessTime": , "blocksize": ,"group":"hdfs","length":19395,"modificationtime": ," owner":"bob","pathsuffix":"readme2","permission":"644","replication":1,"type":"file"] It is a design decision of the DSL to not provide type safe classes for various request and response payloads. Doing so would provide an undesirable coupling between the DSL and the service implementation. It also would make adding new commands much more difficult. See the Groovy section below for a variety capabilities and tools for working with JSON and XML to make this easy. The example below shows the use of JsonSlurper and GPath to extract content from a JSON response. knox:000> import groovy.json.jsonslurper knox:000> text = Hdfs.ls( hadoop ).dir( "/tmp/example" ).now().string knox:000> json = (new JsonSlurper()).parseText( text ) knox:000> println json.filestatuses.filestatus.pathsuffix [README, README2] In the future, "built-in" methods to slurp JSON and XML may be added to make this a bit easier. This would allow for this type if single line interaction. println Hdfs.ls(hadoop).dir("/tmp").now().json().FileStatuses.FileStatus.pathSuffix Shell session should always be ended with shutting down the session. The examples above do not touch on it but the DSL supports the simple execution of commands asynchronously. The shutdown command attempts to ensures that all asynchronous commands have completed before existing the shell. knox:000> hadoop.shutdown() knox:000> exit All of the commands above could have been combined into a script file and executed as a single line. java -jar bin/shell.jar samples/putfile.groovy This script file is available in the distribution but for convenience, this is the content.
4 import org.apache.hadoop.gateway.shell.hadoop import org.apache.hadoop.gateway.shell.hdfs.hdfs import groovy.json.jsonslurper gateway = " username = "bob" password = "bob-password" datafile = "README" hadoop = Hadoop.login( gateway, username, password ) Hdfs.rm( hadoop ).file( "/tmp/example" ).recursive().now() Hdfs.put( hadoop ).file( datafile ).to( "/tmp/example/readme" ).now() text = Hdfs.ls( hadoop ).dir( "/tmp/example" ).now().string json = (new JsonSlurper()).parseText( text ) println json.filestatuses.filestatus.pathsuffix hadoop.shutdown() exit Notice the Hdfs.rm command. This is included simply to ensure that the script can be rerun. Without this an error would result the second time it is run. Futures The DSL supports the ability to invoke commands asynchronously via the later() invocation method. The object returned from the later() method is a java.util.concurrent.future parametrized with the response type of the command. This is an example of how to asynchronously put a file to HDFS. future = Hdfs.put(hadoop).file("README").to("tmp/example/README").later() println future.get().statuscode The future.get() method will block until the asynchronous command is complete. To illustrate the usefullness of this however multiple concurrent commands are required. readmefuture = Hdfs.put(hadoop).file("README").to("tmp/example/README").later() licensefuture = Hdfs.put(hadoop).file("LICENSE").to("tmp/example/LICENSE").later() hadoop.waitfor( readmefuture, licensefuture ) println readmefuture.get().statuscode println licensefuture.get().statuscode The hadoop.waitfor() method will wait for one or more asynchronous commands to complete. Closures Futures alone only provide asynchronous invocation of the command. What if some processing should also occur asynchronously once the command is complete. Support for this is provided by closures. Closures are blocks of code that are passed into the later() invocation method. In Groovy these are contained within { immediately after a method. These blocks of code are executed once the asynchronous command is complete.
5 Hdfs.put(hadoop).file("README").to("tmp/example/README").later(){ println it.statuscode In this example the put() command is executed on a separate thread and once complete the println it.statuscode block is executed on that thread. The it variable is automatically populated by Groovy and is a reference to the result that is returned from the future or now() method. The future example above can be rewritten to illustrate the use of closures. readmefuture = Hdfs.put(hadoop).file("README").to("tmp/example/README").later() { println it.statuscode licensefuture = Hdfs.put(hadoop).file("LICENSE").to("tmp/example/LICENSE").later() { println it.statuscode hadoop.waitfor( readmefuture, licensefuture ) Again, the hadoop.waitfor() method will wait for one or more asynchronous commands to complete. Constructs In order to understand the DSL there are three primary constructs that need to be understood. Hadoop This construct encapsulates the client side session state that will be shared between all command invocations. In particular it will simplify the management of any tokens that need to be presented with each command invocation. It also manages a thread pool that is used by all asynchronous commands which is why it is important to call one of the shutdown methods. The syntax associated with this is expected to change we expect that credentials will not need to be provided to the gateway. Rather it is expected that some form of access token will be used to initialize the session. Services Services are the primary extension point for adding new suites of commands. The built in examples are: Hdfs, Job and Workflow. The desire for extensibility is the reason for the slightly awkward Hdfs.ls(hadoop) syntax. Certainly something more like hadoop.hdfs().ls() would have been preferred but this would prevent adding new commands easily. At a minimum it would result in extension commands with a different syntax from the "built-in" commands. The service objects essentially function as a factory for a suite of commands. Commands Commands provide the behavior of the DSL. They typically follow a Fluent interface style in order to allow for single line commands. There are really three parts to each command:, Invocation, The request is populated by all of the methods following the "verb" method and the "invoke" method. For example in Hdfs.rm(hadoop).ls(dir).now() the request is populated between the "verb" method rm() and the "invoke" method now(). Invocation The invocation method controls how the request is invoked. Currently supported synchronous and asynchronous invocation.
6 The now() method executes the request and returns the result immediately. The later() method submits the request to be executed later and returns a future from which the result can be retrieved. In addition later() invocation method can optionally be provided a closure to execute when the request is complete. See the Futures and Closures sections below for additional detail and examples. The response contains the results of the invocation of the request. In most cases the response is a thin wrapper over the HTTP response. In fact many commands will share a single Basic type that only provides a few simple methods. public int getstatuscode() public long getcontentlength() public String getcontenttype() public String getcontentencoding() public InputStream getstream() public String getstring() public byte[] getbytes() public void close(); Thanks to Groovy these methods can be accessed as attributes. In the some of the examples the staticcode was retrieved for example. println Hdfs.put(hadoop).rm(dir).now().statusCode Groovy will invoke the getstatuscode method to retrieve the statuscode attribute. The three methods getstream(), getbytes() and getstring deserve special attention. Care must be taken that the HTTP body is read only once. Therefore one of these methods (and only one) must be called once and only once. Calling one of these more than once will cause an error. Failing to call one of these methods once will result in lingering open HTTP connections. The close() method may be used if the caller is not interested in reading the result body. Most commands that do not expect a response body will call close implicitly. If the body is retrieved via getbytes() or getstring(), the close() method need not be called. When using getstream(), care must be taken to consume the entire body otherwise lingering open HTTP connections will result. The close() method may be called after reading the body partially to discard the remainder of the body. Services There are three basic DSL services and commands bundled with the shell. HDFS Provides basic HDFS commands. Using these DSL commands requires that WebHDFS be running in the Hadoop cluster. Jobs (Templeton/WebHCat) Provides basic job submission and status commands. Using these DSL commands requires that Templeton/WebHCat be running in the Hadoop cluster. Workflow (Oozie) Provides basic workflow submission and status commands.
7 Using these DSL commands requires that Oozie be running in the Hadoop cluster. HDFS Commands (WebHDFS) ls() - List the contents of a HDFS directory. dir (String) - The HDFS directory to list. Basic Hdfs.ls(hadoop).ls().dir("/").now() rm() - Remove a HDFS file or directory. file (String) - The HDFS file or directory to remove. recursive (Boolean) - If the file is a directory also remove any contained files and directories. Optional: default=false Empty - Implicit close(). Hdfs.rm(hadoop).file("/tmp/example").recursive().now() put() - Copy a file from the local file system to HDFS. text (String) - The text to copy to the remote file. file (String) - The name of a local file to copy to the remote file. to (String) - The name of the remote file create. Empty - Implicit close(). Hdfs.put(hadoop).file("localFile").to("/tmp/example/remoteFile").now() get() - Copy a file from HDFS to the local file system. file (String) - The name of the local file to create from the remote file. If this isn't specified the file content must be read from the response. from (String) - The name of the remote file to copy. Basic Hdfs.get(hadoop).file("localFile").from("/tmp/example/remoteFile").now() mkdir() - Create a directory in HDFS. dir (String) - The name of the remote directory to create. perm (String) - The permissions to create the remote directory with. Optional: default="777" Empty - Implicit close(). Hdfs.mkdir(hadoop).dir("/tmp/example").perm("777").now() Job Commands (WebHCat/Templeton) submitjava() - Submit a Java MapReduce job.
8 jar (String) - The remote file name of the JAR containing the app to execute. app (String) - The app name to execute. This is wordcount for example not the class name. input (String) - The remote directory name to use as input for the job. output (String) - The remote directory name to store output from the job. jobid : String - The job ID of the submitted job. Consumes body. Job.submitJava(hadoop).jar(remoteJarName).app(appName).input(remoteInputDir).output(remoteOutpu tdir).now().jobid submitpig() - Submit a Pig job. file (String) - The remote file name of the pig script. arg (String) - An argument to pass to the script. statusdir (String) - The remote directory to store status output. jobid : String - The job ID of the submitted job. Consumes body. Job.submitPig(hadoop).file(remotePigFileName).arg("-v").statusDir(remoteStatusDir).now() submithive() - Submit a Hive job. file (String) - The remote file name of the hive script. arg (String) - An argument to pass to the script. statusdir (String) - The remote directory to store status output. jobid : String - The job ID of the submitted job. Consumes body. Job.submitHive(hadoop).file(remoteHiveFileName).arg("-v").statusDir(remoteStatusDir).now() queryqueue() - Return a list of all job IDs registered to the user. No request parameters. Basic Job.queryQueue(hadoop).now().string querystatus() - Check the status of a job and get related job information given its job ID. jobid (String) - The job ID to check. This is the ID received when the job was created. Basic Job.queryStatus(hadoop).jobId(jobId).now().string Workflow Commands (Oozie) submit() - Submit a workflow job. text (String) - XML formatted workflow configuration string. file (String) - A filename containing XML formatted workflow configuration. action (String) - The initial action to take on the job. Optional: Default is "start". Basic Workflow.submit(hadoop).file(localFile).action("start").now()
9 status() - Query the status of a workflow job. jobid (String) - The job ID to check. This is the ID received when the job was created. Basic Workflow.status(hadoop).jobId(jobId).now().string Extension Extensibility is a key design goal of the KnoxShell and DSL. There are two ways to provide extended functionality for use with the shell. The first is to simply create Groovy scripts that use the DSL to perform a useful task. The second is to add new services and commands. In order to add new service and commands new classes must be written in either Groovy or Java and added to the classpath of the shell. Fortunately there is a very simple way to add classes and JARs to the shell classpath. The first time the shell is executed it will create a configuration file in the same directory as the JAR with the same base name and a.cfg on. extensi bin/shell.jar bin/shell.cfg That file contains both the main class for the shell as well as a definition of the classpath. Currently that file will by default contain the following. main.class=org.apache.hadoop.gateway.shell.shell class.path=../lib;../lib/*.jar;../ext;../ext/*.jar Therefore to extend the shell you should copy any new service and command class either to the ext directory or if they are packaged within a JAR copy the JAR to the ext directory. The lib directory is reserved for JARs that may be delivered with the product. Below are samples for the service and command classes that would need to be written to add new commands to the shell. These happen to be Groovy source files but could with very minor changes be Java files. The easiest way to add these to the shell is to compile them directory into the ext directory. Note: This command depends upon having the Groovy compiler installed and available on the execution path. groovyc \-d ext \-cp bin/shell.jar samples/sampleservice.groovy samples/samplesimplecommand.groovy samples/samplecomplexcommand.groovy These source files are available in the samples directory of the distribution but these are included here for convenience. Sample Service (Groovy)
10 import org.apache.hadoop.gateway.shell.hadoop class SampleService { static String PATH = "/namenode/api/v1" static SimpleCommand simple( Hadoop hadoop ) { return new SimpleCommand( hadoop ) static ComplexCommand. complex( Hadoop hadoop ) { return new ComplexCommand.( hadoop ) Sample Simple Command (Groovy)
11 import org.apache.hadoop.gateway.shell.abstract import org.apache.hadoop.gateway.shell.basic import org.apache.hadoop.gateway.shell.hadoop import org.apache.http.client.methods.httpget import org.apache.http.client.utils.uribuilder import java.util.concurrent.callable class SimpleCommand extends Abstract<Basic> { SimpleCommand( Hadoop hadoop ) { super( hadoop ) private String param SimpleCommand param( String param ) { this.param = param return protected Callable<Basic> callable() { return new Callable<Basic>() Basic call() { URIBuilder uri = uri( SampleService.PATH, param ) addqueryparam( uri, "op", "LISTSTATUS" ) HttpGet get = new HttpGet( uri.build() ) return new Basic( execute( get ) ) Sample Complex Command (Groovy)
12 import com.jayway.jsonpath.jsonpath import org.apache.hadoop.gateway.shell.abstract import org.apache.hadoop.gateway.shell.basic import org.apache.hadoop.gateway.shell.hadoop import org.apache.http.http import org.apache.http.client.methods.httpget import org.apache.http.client.utils.uribuilder import java.util.concurrent.callable class ComplexCommand { static class extends Abstract<> { ( Hadoop hadoop ) { super( hadoop ) private String param; param( String param ) { this.param = param; return protected Callable<> callable() { return new Callable<>() call() { URIBuilder uri = uri( SampleService.PATH, param ) addqueryparam( uri, "op", "LISTSTATUS" ) HttpGet get = new HttpGet( uri.build() ) return new ( execute( get ) ) static class extends Basic { (Http response) { super(response) public List<String> getnames() { return JsonPath.read( string, "\$.FileStatuses.FileStatus[*].pathSuffix" )
13 Groovy The shell included in the distribution is basically an unmodified packaging of the Groovy shell. The distribution does however provide a wrapper that makes it very easy to setup the class path for the shell. In fact the JARs required to execute the DSL are included on the class path by default. Therefore these command are functionally equivalent if you have Groovy installed15. See below for a description of what is required for { JARs required by the DSL from lib and dep java -jar bin/shell.jar samples/putfile.groovy groovy -classpath {JARs required by the DSL from lib and dep samples/putfile.groovy The interactive shell isn't exactly equivalent. However the only difference is that the shell.jar automatically executes some additional imports that are useful for the KnoxShell DSL. So these two sets of commands should be functionality equivalent. However there is currently a class loading issue that prevents the groovysh command from working propertly. java -jar bin/shell.jar groovysh -classpath {JARs required by the DSL from lib and dep import org.apache.hadoop.gateway.shell.hadoop import org.apache.hadoop.gateway.shell.hdfs.hdfs import org.apache.hadoop.gateway.shell.job.job import org.apache.hadoop.gateway.shell.workflow.workflow import java.util.concurrent.timeunit Alternatively, you can use the Groovy Console which does not appear to have the same class loading issue. groovyconsole -classpath {JARs required by the DSL from lib and dep import org.apache.hadoop.gateway.shell.hadoop import org.apache.hadoop.gateway.shell.hdfs.hdfs import org.apache.hadoop.gateway.shell.job.job import org.apache.hadoop.gateway.shell.workflow.workflow import java.util.concurrent.timeunit The list of JARs currently required by the DSL is lib/gateway-shell-${gateway-version.jar dep/httpclient jar dep/httpcore jar dep/commons-lang3-3.1.jar dep/commons-codec-1.7.jar So on Linux/MacOS you would need this command
14 groovy -cp lib/gateway-shell snapshot.jar:dep/httpclient jar:dep/httpcore jar:d ep/commons-lang3-3.1.jar:dep/commons-codec-1.7.jar samples/putfile.groovy and on Windows you would need this command groovy -cp lib/gateway-shell snapshot.jar;dep/httpclient jar;dep/httpcore jar;d ep/commons-lang3-3.1.jar;dep/commons-codec-1.7.jar samples/putfile.groovy The exact list of required JARs is likely to change from release to release so it is recommended that you utilize the wrapper bin/shell.jar. In addition because the DSL can be used via standard Groovy, the Groovy integrations in many popular IDEs (e.g. IntelliJ, Eclipse) can also be used. This makes it particularly nice to develop and execute scripts to interact with Hadoop. The code-completion feature in particular provides immense value. All that is required is to add the shell jar to the projects class path. There are a variety of Groovy tools that make it very easy to work with the standard interchange formats (i.e. JSON and XML). In Groovy the creation of XML or JSON is typically done via a "builder" and parsing done via a "slurper". In addition once JSON or XML is "slurped" the GPath, an XPath like feature build into Groovy can be used to access data. XML JSON Markup Builder Overview ( API XML Slurper Overview ( API XPath Overview, API JSON Builder API JSON Slurper API JSON Path API GPath Overview Trademarks Apache Knox Gateway, Apache, the Apache feather logo and the Apache Knox Gateway project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners. License Apache Knox uses the standard Apache license. Privacy Policy Apache Knox uses the standard Apache privacy policy.
SQL Client Example using KnoxShell in Apache Knox
SQL Client Example using KnoxShell in Apache Knox The KnoxShell release artifact provides a small footprint client environment that removes all unnecessary server dependencies, configuration, binary scripts,
More informationHortonworks Data Platform
Hortonworks Data Platform Workflow Management (August 31, 2017) docs.hortonworks.com Hortonworks Data Platform: Workflow Management Copyright 2012-2017 Hortonworks, Inc. Some rights reserved. The Hortonworks
More informationChapter 4 Defining Classes I
Chapter 4 Defining Classes I This chapter introduces the idea that students can create their own classes and therefore their own objects. Introduced is the idea of methods and instance variables as the
More informationGRID COMPANION GUIDE
Companion Subject: GRID COMPANION Author(s): Miguel Cárdenas Montes, Antonio Gómez Iglesias, Francisco Castejón, Adrian Jackson, Joachim Hein Distribution: Public 1.Introduction Here you will find the
More informationConfiguring Apache Knox SSO
3 Configuring Apache Knox SSO Date of Publish: 2018-07-15 http://docs.hortonworks.com Contents Setting Up Knox SSO...3 Configuring an Identity Provider (IdP)... 3 Configuring an LDAP/AD Identity Provider
More informationCentrify for Dropbox Deployment Guide
CENTRIFY DEPLOYMENT GUIDE Centrify for Dropbox Deployment Guide Abstract Centrify provides mobile device management and single sign-on services that you can trust and count on as a critical component of
More informationSetting Up Resources in VMware Identity Manager (On Premises) Modified on 30 AUG 2017 VMware AirWatch 9.1.1
Setting Up Resources in VMware Identity Manager (On Premises) Modified on 30 AUG 2017 VMware AirWatch 9.1.1 Setting Up Resources in VMware Identity Manager (On Premises) You can find the most up-to-date
More informationSetting Up Resources in VMware Identity Manager. VMware Identity Manager 2.8
Setting Up Resources in VMware Identity Manager VMware Identity Manager 2.8 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments
More informationDCLI User's Guide. Data Center Command-Line Interface 2.9.1
Data Center Command-Line Interface 2.9.1 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about this documentation, submit
More informationConfiguring Apache Knox SSO
3 Configuring Apache Knox SSO Date of Publish: 2018-07-15 http://docs.hortonworks.com Contents Configuring Knox SSO... 3 Configuring an Identity Provider (IdP)... 4 Configuring an LDAP/AD Identity Provider
More informationSetting Up Resources in VMware Identity Manager (SaaS) Modified 15 SEP 2017 VMware Identity Manager
Setting Up Resources in VMware Identity Manager (SaaS) Modified 15 SEP 2017 VMware Identity Manager Setting Up Resources in VMware Identity Manager (SaaS) You can find the most up-to-date technical documentation
More informationServer Installation Guide
Server Installation Guide Server Installation Guide Legal notice Copyright 2018 LAVASTORM ANALYTICS, INC. ALL RIGHTS RESERVED. THIS DOCUMENT OR PARTS HEREOF MAY NOT BE REPRODUCED OR DISTRIBUTED IN ANY
More informationAbout 1. Chapter 1: Getting started with oozie 2. Remarks 2. Versions 2. Examples 2. Installation or Setup 2. Chapter 2: Oozie
oozie #oozie Table of Contents About 1 Chapter 1: Getting started with oozie 2 Remarks 2 Versions 2 Examples 2 Installation or Setup 2 Chapter 2: Oozie 101 7 Examples 7 Oozie Architecture 7 Oozie Application
More informationData Avenue REST API. Ákos Hajnal, Zoltán Farkas November, 2015
Data Avenue REST API Ákos Hajnal, Zoltán Farkas November, 2015 What is REST? REST (Representational State Transfer) is an architectural style (Roy Fielding, 2000) client-server model, stateless (individually
More informationPace University. Fundamental Concepts of CS121 1
Pace University Fundamental Concepts of CS121 1 Dr. Lixin Tao http://csis.pace.edu/~lixin Computer Science Department Pace University October 12, 2005 This document complements my tutorial Introduction
More informationDCLI User's Guide. Data Center Command-Line Interface
Data Center Command-Line Interface 2.10.2 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about this documentation, submit
More informationDCLI User's Guide. Data Center Command-Line Interface 2.7.0
Data Center Command-Line Interface 2.7.0 You can find the most up-to-date technical documentation on the VMware Web site at: https://docs.vmware.com/ The VMware Web site also provides the latest product
More informationDCLI User's Guide. Modified on 20 SEP 2018 Data Center Command-Line Interface
Modified on 20 SEP 2018 Data Center Command-Line Interface 2.10.0 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about
More informationAdvanced Topics in WebSphere Portal Development Graham Harper Application Architect IBM Software Services for Collaboration
Advanced Topics in WebSphere Portal Development Graham Harper Application Architect IBM Software Services for Collaboration 2012 IBM Corporation Ideas behind this session Broaden the discussion when considering
More informationMake sure you have the latest Hive trunk by running svn up in your Hive directory. More detailed instructions on downloading and setting up
GenericUDAFCaseStudy Writing GenericUDAFs: A Tutorial User-Defined Aggregation Functions (UDAFs) are an excellent way to integrate advanced data-processing into Hive. Hive allows two varieties of UDAFs:
More informationOperating Systems. Copyleft 2005, Binnur Kurt
3 Operating Systems Copyleft 2005, Binnur Kurt Content The concept of an operating system. The internal architecture of an operating system. The architecture of the Linux operating system in more detail.
More informationOperating Systems 3. Operating Systems. Content. What is an Operating System? What is an Operating System? Resource Abstraction and Sharing
Content 3 Operating Systems The concept of an operating system. The internal architecture of an operating system. The architecture of the Linux operating system in more detail. How to log into (and out
More informationBig Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours
Big Data Hadoop Developer Course Content Who is the target audience? Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Complete beginners who want to learn Big Data Hadoop Professionals
More informationGetting. Started with. smash. IBM WebSphere. Ron Lynn, Karl Bishop, Brett King
Getting Started with IBM WebSphere smash Ron Lynn, Karl Bishop, Brett King Contents Introduction 1 Situational Applications 1 Rapid Application Development 1 IBM WebSphere smash Development Process 2 Available
More informationGroovy For Java Programmers
Groovy For Java Programmers QCONSF 2010 Jeff Brown Core Grails Developer jeff.brown@springsource.com SpringSource - A Division Of VMware http://springsource.com/ Copyright 2010 SpringSource. Copying, publishing
More informationSetting Up Resources in VMware Identity Manager
Setting Up Resources in VMware Identity Manager VMware Identity Manager 2.7 This document supports the version of each product listed and supports all subsequent versions until the document is replaced
More informationHands-on Exercise Hadoop
Department of Economics and Business Administration Chair of Business Information Systems I Prof. Dr. Barbara Dinter Big Data Management Hands-on Exercise Hadoop Building and Testing a Hadoop Cluster by
More informationScheduling in SAS 9.4, Second Edition
Scheduling in SAS 9.4, Second Edition SAS Documentation September 5, 2017 The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2016. Scheduling in SAS 9.4, Second Edition.
More informationShell and Utility Commands
Table of contents 1 Shell Commands... 2 2 Utility Commands... 3 1 Shell Commands 1.1 fs Invokes any FsShell command from within a Pig script or the Grunt shell. 1.1.1 Syntax fs subcommand subcommand_parameters
More informationCloudera Connector for Netezza
Cloudera Connector for Netezza Important Notice 2010-2017 Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, and any other product or service names or slogans contained in this document are
More informationActiveSpaces Transactions. Quick Start Guide. Software Release Published May 25, 2015
ActiveSpaces Transactions Quick Start Guide Software Release 2.5.0 Published May 25, 2015 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED
More informationSandbox Setup Guide for HDP 2.2 and VMware
Waterline Data Inventory Sandbox Setup Guide for HDP 2.2 and VMware Product Version 2.0 Document Version 10.15.2015 2014-2015 Waterline Data, Inc. All rights reserved. All other trademarks are the property
More informationHadoop File System Commands Guide
Hadoop File System Commands Guide (Learn more: http://viewcolleges.com/online-training ) Table of contents 1 Overview... 3 1.1 Generic Options... 3 2 User Commands...4 2.1 archive...4 2.2 distcp...4 2.3
More informationParallel Programming Pre-Assignment. Setting up the Software Environment
Parallel Programming Pre-Assignment Setting up the Software Environment Authors: B. Wilkinson and C. Ferner. Modification date: Aug 21, 2014 (Minor correction Aug 27, 2014.) Software The purpose of this
More informationIntroduction to application management
Introduction to application management To deploy web and mobile applications, add the application from the Centrify App Catalog, modify the application settings, and assign roles to the application to
More informationKnox Implementation with AD/LDAP
Knox Implementation with AD/LDAP Theory part Introduction REST API and Application Gateway for the Apache Hadoop Ecosystem: The Apache Knox Gateway is an Application Gateway for interacting with the REST
More information$HIVE_HOME/bin/hive is a shell utility which can be used to run Hive queries in either interactive or batch mode.
LanguageManual Cli Hive CLI Hive CLI Deprecation in favor of Beeline CLI Hive Command Line Options Examples The hiverc File Logging Tool to Clear Dangling Scratch Directories Hive Batch Mode Commands Hive
More informationCreating WebLogic Domains Using the Configuration Wizard 12c (12.1.3)
[1]Oracle Fusion Middleware Creating WebLogic 12.1.3 Domains Using the Configuration Wizard 12c (12.1.3) E41890-02 August 2015 This document describes how to use the Configuration Wizard to create, update,
More informationHortonworks Technical Preview for Apache Falcon
Architecting the Future of Big Data Hortonworks Technical Preview for Apache Falcon Released: 11/20/2013 Architecting the Future of Big Data 2013 Hortonworks Inc. All Rights Reserved. Welcome to Hortonworks
More informationAventail WorkPlace. User s Guide Version 8.7.0
Aventail WorkPlace User s Guide Version 8.7.0 1996-2006 Aventail Corporation. All rights reserved. Aventail, Aventail Cache Control, Aventail Connect, Aventail Connect Mobile, Aventail Connect Tunnel,
More informationVMware vsphere Big Data Extensions Administrator's and User's Guide
VMware vsphere Big Data Extensions Administrator's and User's Guide vsphere Big Data Extensions 1.1 This document supports the version of each product listed and supports all subsequent versions until
More informationHortonworks HDPCD. Hortonworks Data Platform Certified Developer. Download Full Version :
Hortonworks HDPCD Hortonworks Data Platform Certified Developer Download Full Version : https://killexams.com/pass4sure/exam-detail/hdpcd QUESTION: 97 You write MapReduce job to process 100 files in HDFS.
More informationOracle Fusion Middleware
Oracle Fusion Middleware Creating Domains Using the Configuration Wizard 11g Release 1 (10.3.4) E14140-04 January 2011 This document describes how to use the Configuration Wizard to create, update, and
More informationHadoop On Demand User Guide
Table of contents 1 Introduction...3 2 Getting Started Using HOD... 3 2.1 A typical HOD session... 3 2.2 Running hadoop scripts using HOD...5 3 HOD Features... 6 3.1 Provisioning and Managing Hadoop Clusters...6
More informationA Guide to Running Map Reduce Jobs in Java University of Stirling, Computing Science
A Guide to Running Map Reduce Jobs in Java University of Stirling, Computing Science Introduction The Hadoop cluster in Computing Science at Stirling allows users with a valid user account to submit and
More informationGetting Started with the Bullhorn SOAP API and Java
Getting Started with the Bullhorn SOAP API and Java Introduction This article is targeted at developers who want to do custom development using the Bullhorn SOAP API and Java. You will create a sample
More informationHadoop-PR Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer)
Hortonworks Hadoop-PR000007 Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer) http://killexams.com/pass4sure/exam-detail/hadoop-pr000007 QUESTION: 99 Which one of the following
More informationCloudera Manager Quick Start Guide
Cloudera Manager Guide Important Notice (c) 2010-2015 Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, Cloudera Impala, and any other product or service names or slogans contained in this
More informationIntroduction to Cloudbreak
2 Introduction to Cloudbreak Date of Publish: 2019-02-06 https://docs.hortonworks.com/ Contents What is Cloudbreak... 3 Primary use cases... 3 Interfaces...3 Core concepts... 4 Architecture... 7 Cloudbreak
More informationCS 6353 Compiler Construction Project Assignments
CS 6353 Compiler Construction Project Assignments In this project, you need to implement a compiler for a language defined in this handout. The programming language you need to use is C or C++ (and the
More informationΠΙΝΑΚΑΣ ΠΛΑΝΟΥ ΕΚΠΑΙΔΕΥΣΗΣ
ΠΑΡΑΡΤΗΜΑ «Β» ΠΙΝΑΚΑΣ ΠΛΑΝΟΥ ΕΚΠΑΙΔΕΥΣΗΣ Α/Α ΠΕΡΙΓΡΑΦΗ ΕΚΠΑΙΔΕΥΣΗΣ ΘΕΜΑΤΙΚΕΣ ΕΝΟΤΗΤΕΣ 1. Java SE8 Fundamentals What Is a Java Program? Introduction to Computer Programs Key Features of the Java Language
More informationHadoop On Demand: Configuration Guide
Hadoop On Demand: Configuration Guide Table of contents 1 1. Introduction...2 2 2. Sections... 2 3 3. HOD Configuration Options...2 3.1 3.1 Common configuration options...2 3.2 3.2 hod options... 3 3.3
More informationHarnessing the Power of YARN with Apache Twill
Harnessing the Power of YARN with Apache Twill Andreas Neumann andreas[at]continuuity.com @anew68 A Distributed App Reducers part part part shuffle Mappers split split split A Map/Reduce Cluster part
More informationLogging on to the Hadoop Cluster Nodes. To login to the Hadoop cluster in ROGER, a user needs to login to ROGER first, for example:
Hadoop User Guide Logging on to the Hadoop Cluster Nodes To login to the Hadoop cluster in ROGER, a user needs to login to ROGER first, for example: ssh username@roger-login.ncsa. illinois.edu after entering
More informationIbis RMI User s Guide
Ibis RMI User s Guide http://www.cs.vu.nl/ibis November 16, 2009 1 Introduction Java applications typically consist of one or more threads that manipulate a collection of objects by invoking methods on
More informationLab Compiling using an IDE (Eclipse)
Lab 1. This introductory lab is composed of three tasks. Your final objective is to run your first Hadoop application. For this goal, you must learn how to compile the source code and produce a jar, connect
More informationHOD User Guide. Table of contents
Table of contents 1 Introduction...3 2 Getting Started Using HOD... 3 2.1 A typical HOD session... 3 2.2 Running hadoop scripts using HOD...5 3 HOD Features... 6 3.1 Provisioning and Managing Hadoop Clusters...6
More informationUser Plugins. About Plugins. Deploying Plugins
User Plugins About Plugins Artifactory Pro allows you to easily extend Artifactory's behavior with your own plugins written in Groovy. User plugins are used for running user's code in Artifactory. Plugins
More informationUsing Hive for Data Warehousing
An IBM Proof of Technology Using Hive for Data Warehousing Unit 1: Exploring Hive An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2013 US Government Users Restricted Rights - Use,
More informationArgos Version 3.1 Release Guide, Version 1.0 Last Updated 04/04/2008 TABLE OF CONTENTS. Introduction MAP Server Updates... 4
Argos Version 3.1 Release Guide Document version 1.0 TABLE OF CONTENTS Introduction... 3 MAP Server Updates... 4 Thread Lock Fix... 4 Individual Processes for Each Scheduled Report... 4 Automatic Termination
More informationVMware vfabric AppInsight Installation Guide
VMware vfabric AppInsight Installation Guide vfabric AppInsight 5.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new
More informationSAS 9.2 Foundation Services. Administrator s Guide
SAS 9.2 Foundation Services Administrator s Guide The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2009. SAS 9.2 Foundation Services: Administrator s Guide. Cary, NC:
More informationDOCS
HOME DOWNLOAD COMMUNITY DEVELOP NEWS DOCS Docker Images Docker Images for Avatica Docker is a popular piece of software that enables other software to run anywhere. In the context of Avatica, we can use
More informationCompaq Interview Questions And Answers
Part A: Q1. What are the difference between java and C++? Java adopts byte code whereas C++ does not C++ supports destructor whereas java does not support. Multiple inheritance possible in C++ but not
More informationContents Overview... 5 Upgrading Primavera Gateway... 7 Using Gateway Configuration Utilities... 9
Gateway Upgrade Guide for On-Premises Version 17 August 2017 Contents Overview... 5 Downloading Primavera Gateway... 5 Upgrading Primavera Gateway... 7 Prerequisites... 7 Upgrading Existing Gateway Database...
More informationShell and Utility Commands
Table of contents 1 Shell Commands... 2 2 Utility Commands...3 1. Shell Commands 1.1. fs Invokes any FsShell command from within a Pig script or the Grunt shell. 1.1.1. Syntax fs subcommand subcommand_parameters
More informationInstalling Apache Knox
3 Installing Apache Knox Date of Publish: 2018-07-15 http://docs.hortonworks.com Contents...3 Install Knox...3 Set Up Knox Proxy... 4 Example: Configure Knox Gateway for YARN UI...6 Example: Configure
More informationGuided Tour (Version 3.4) By Steven Castellucci
Guided Tour (Version 3.4) By Steven Castellucci This document was inspired by the Guided Tour written by Professor H. Roumani. His version of the tour can be accessed at the following URL: http://www.cse.yorku.ca/~roumani/jbayork/guidedtour.pdf.
More informationVMware vsphere Big Data Extensions Command-Line Interface Guide
VMware vsphere Big Data Extensions Command-Line Interface Guide vsphere Big Data Extensions 2.0 This document supports the version of each product listed and supports all subsequent versions until the
More informationWeek 2 Lecture 3. Unix
Lecture 3 Unix Terminal and Shell 2 Terminal Prompt Command Argument Result 3 Shell Intro A system program that allows a user to execute: shell functions (e.g., ls -la) other programs (e.g., eclipse) shell
More informationIntroduction to UNIX. Logging in. Basic System Architecture 10/7/10. most systems have graphical login on Linux machines
Introduction to UNIX Logging in Basic system architecture Getting help Intro to shell (tcsh) Basic UNIX File Maintenance Intro to emacs I/O Redirection Shell scripts Logging in most systems have graphical
More informationOpenIAM Identity and Access Manager Technical Architecture Overview
OpenIAM Identity and Access Manager Technical Architecture Overview Overview... 3 Architecture... 3 Common Use Case Description... 3 Identity and Access Middleware... 5 Enterprise Service Bus (ESB)...
More informationOracle Application Express: Administration 1-2
Oracle Application Express: Administration 1-2 The suggested course agenda is displayed in the slide. Each lesson, except the Course Overview, will be followed by practice time. Oracle Application Express:
More informationProblem Set 0. General Instructions
CS246: Mining Massive Datasets Winter 2014 Problem Set 0 Due 9:30am January 14, 2014 General Instructions This homework is to be completed individually (no collaboration is allowed). Also, you are not
More informationSAS Viya 3.2 and SAS/ACCESS : Hadoop Configuration Guide
SAS Viya 3.2 and SAS/ACCESS : Hadoop Configuration Guide SAS Documentation July 6, 2017 The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2017. SAS Viya 3.2 and SAS/ACCESS
More informationLeveraging the Globus Platform in your Web Applications. GlobusWorld April 26, 2018 Greg Nawrocki
Leveraging the Globus Platform in your Web Applications GlobusWorld April 26, 2018 Greg Nawrocki greg@globus.org Topics and Goals Platform Overview Why expose the APIs A quick touch of the Globus Auth
More informationJava Cookbook. Java Action specification. $ java -Xms512m a.b.c.mymainclass arg1 arg2
Java Cookbook This document comprehensively describes the procedure of running Java code using Oozie. Its targeted audience is all forms of users who will install, use and operate Oozie. Java Action specification
More informationVMware AirWatch Integration with F5 Guide Enabling secure connections between mobile applications and your backend resources
VMware AirWatch Integration with F5 Guide Enabling secure connections between mobile applications and your backend resources Workspace ONE UEM v9.6 Have documentation feedback? Submit a Documentation Feedback
More informationHDP Security Overview
3 HDP Security Overview Date of Publish: 2018-07-15 http://docs.hortonworks.com Contents HDP Security Overview...3 Understanding Data Lake Security... 3 What's New in This Release: Knox... 5 What's New
More informationHDP Security Overview
3 HDP Security Overview Date of Publish: 2018-07-15 http://docs.hortonworks.com Contents HDP Security Overview...3 Understanding Data Lake Security... 3 What's New in This Release: Knox... 5 What's New
More informationIntroduction to Hadoop. High Availability Scaling Advantages and Challenges. Introduction to Big Data
Introduction to Hadoop High Availability Scaling Advantages and Challenges Introduction to Big Data What is Big data Big Data opportunities Big Data Challenges Characteristics of Big data Introduction
More informationAPI Knowledge Coding Guide Version 7.2
API Knowledge Coding Guide Version 7.2 You will be presented with documentation blocks extracted from API reference documentation (Javadocs and the like). For each block, you will be also presented with
More informationCreating Domain Templates Using the Domain Template Builder 11g Release 1 (10.3.6)
[1]Oracle Fusion Middleware Creating Domain Templates Using the Domain Template Builder 11g Release 1 (10.3.6) E14139-06 April 2015 This document describes how to use the Domain Template Builder to create
More informationGlobalbrain Administration Guide. Version 5.4
Globalbrain Administration Guide Version 5.4 Copyright 2012 by Brainware, Inc. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system,
More informationGuided Tour (Version 3.3) By Steven Castellucci as Modified by Brandon Haworth
Guided Tour (Version 3.3) By Steven Castellucci as Modified by Brandon Haworth This document was inspired by the Guided Tour written by Professor H. Roumani. His version of the tour can be accessed at
More informationIntroduction to Linux
Introduction to Linux The command-line interface A command-line interface (CLI) is a type of interface, that is, a way to interact with a computer. Window systems, punched cards or a bunch of dials, buttons
More informationJava Programming. Manuel Oriol, March 22nd, 2007
Java Programming Manuel Oriol, March 22nd, 2007 Goal Teach Java to proficient programmers 2 Roadmap Java Basics Eclipse Java GUI Threads and synchronization Class loading and reflection Java Virtual Machines
More informationJava SE7 Fundamentals
Java SE7 Fundamentals Introducing the Java Technology Relating Java with other languages Showing how to download, install, and configure the Java environment on a Windows system. Describing the various
More informationHadoop MapReduce Framework
Hadoop MapReduce Framework Contents Hadoop MapReduce Framework Architecture Interaction Diagram of MapReduce Framework (Hadoop 1.0) Interaction Diagram of MapReduce Framework (Hadoop 2.0) Hadoop MapReduce
More informationGroovy. Extending Java with scripting capabilities. Last updated: 10 July 2017
Groovy Extending Java with scripting capabilities Last updated: 10 July 2017 Pepgo Limited, 71-75 Shelton Street, Covent Garden, London, WC2H 9JQ, United Kingdom Contents About Groovy... 3 Install Groovy...
More informationA- Core Java Audience Prerequisites Approach Objectives 1. Introduction
OGIES 6/7 A- Core Java The Core Java segment deals with the basics of Java. It is designed keeping in mind the basics of Java Programming Language that will help new students to understand the Java language,
More informationSAS 9.4 Foundation Services: Administrator s Guide
SAS 9.4 Foundation Services: Administrator s Guide SAS Documentation July 18, 2017 The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2013. SAS 9.4 Foundation Services:
More informationAdobe ColdFusion 11 Enterprise Edition
Adobe ColdFusion 11 Enterprise Edition Version Comparison Adobe ColdFusion 11 Enterprise Edition Adobe ColdFusion 11 Enterprise Edition is an all-in-one application server that offers you a single platform
More informationDistributed Systems Project 4 Assigned: Friday March 20 Due: Friday April 3, 11:59pm
95-702 Distributed Systems Project 4 Assigned: Friday March 20 Due: Friday April 3, 11:59pm Project Topics: Java RMI and a distributed, Mobile to Cloud application This project has 2 tasks. Task 1 is a
More informationExtending The QiQu Script Language
Extending The QiQu Script Language Table of Contents Setting up an Eclipse-Javaproject to extend QiQu...1 Write your first QiQu Command...2 getcommandname...2 getdescription...2 getcommandgroup...2 isusingsubcommand...3
More informationIntroduction of PDE.Mart
Grid-Based PDE.Mart A PDE-Oriented PSE for Grid Computing GY MAO, M. MU, Wu ZHANG, XB ZHANG School of Computer Science and Engineering, Shanghai University, CHINA Department of Mathematics, Hong Kong University
More informationPieces of the puzzle. Wednesday, March 09, :29 PM
SOAP_and_Axis Page 1 Pieces of the puzzle Wednesday, March 09, 2011 12:29 PM Pieces of the puzzle so far Google AppEngine/GWTJ: a platform for cloud computing. Map/Reduce: a core technology of cloud computing.
More informationIgniting QuantLib on a Zeppelin
Igniting QuantLib on a Zeppelin Andreas Pfadler, d-fine GmbH QuantLib UserMeeting, Düsseldorf, 7.12.2016 d-fine d-fine All rights All rights reserved reserved 0 Welcome Back!» An early stage of this work
More informationOracle Cloud Using Oracle Big Data Cloud. Release 18.1
Oracle Cloud Using Oracle Big Data Cloud Release 18.1 E70336-14 March 2018 Oracle Cloud Using Oracle Big Data Cloud, Release 18.1 E70336-14 Copyright 2017, 2018, Oracle and/or its affiliates. All rights
More informationCGI Architecture Diagram. Web browser takes response from web server and displays either the received file or error message.
What is CGI? The Common Gateway Interface (CGI) is a set of standards that define how information is exchanged between the web server and a custom script. is a standard for external gateway programs to
More information