BMC Control-M Workload Automation Embracing Hadoop batch processing within the enterprise

Transcription

1 BMC Control-M Workload Automation Embracing Hadoop batch processing within the enterprise

2 IDC Big Data Technology & Services The Big Data Market $4.8 Billion with a 39.4% CAGR Projected to reach $16.9 Billion in 2015 Software is 29.7% of market 2013 Control-M Customer Survey Implemented Big Data 24% Including in plans 53% 2700 * 0.53 * 170K = $243M 45 Copyright 6/4/2014 BMC Software, Inc Copyright 6/4/2014 BMC Software, Inc 45

3 Copyright 6/4/2014 BMC Software, Inc 46

4 Script for Running Three Step Java MapReduce #!/usr/bin/env bash bin=`dirname "$0"` bin=`cd "$bin"; pwd`. "$bin"/../libexec/hadoop-config.sh #set the hadoop command and the path to the hadoop examples jar HADOOP_CMD="${HADOOP_PREFIX}/bin/hadoop --config $HADOOP_CONF_DIR #find the hadoop examples jar HADOOP_EXAMPLES_JAR=' #find under HADOOP_PREFIX (tar ball install) HADOOP_EXAMPLES_JAR=`find ${HADOOP_PREFIX} -name 'hadoop-examples-*.jar' head -n1` #if its not found look under /usr/share/hadoop (rpm/deb installs) if [ "$HADOOP_EXAMPLES_JAR" == '' ]then HADOOP_EXAMPLES_JAR=`find /usr/share/hadoop -name 'hadoop-examples-*.jar' head -n1` fi #if it is still empty then dont run the tests if [ "$HADOOP_EXAMPLES_JAR" == '' ]then echo "Did not find hadoop-examples-*.jar under '${HADOOP_PREFIX} or '/usr/share/hadoop'" fi #dir where to store the data on hdfs. The data is relative of the users home dir on hdfs. PARENT_DIR="validate_deploy_`date+%s` TERA_GEN_OUTPUT_DIR="${PARENT_DIR}/tera_gen_data TERA_SORT_OUTPUT_DIR="${PARENT_DIR}/tera_sort_data TERA_VALIDATE_OUTPUT_DIR="${PARENT_DIR}/tera_validate_data #tera gen cmd TERA_GEN_CMD="su -c '$HADOOP_CMD jar $HADOOP_EXAMPLES_JAR teragen $TERA_GEN_OUTPUT_DIR' $TEST_USER #tera sort cmd TERA_SORT_CMD="su -c '$HADOOP_CMD jar $HADOOP_EXAMPLES_JAR terasort $TERA_GEN_OUTPUT_DIR $TERA_SORT_OUTPUT_DIR' $TEST_USER #tera validate cmd TERA_VALIDATE_CMD="su -c '$HADOOP_CMD jar $HADOOP_EXAMPLES_JAR teravalidate $TERA_SORT_OUTPUT_DIR $TERA_VALIDATE_OUTPUT_DIR' $TEST_USER echo "Starting teragen... #run tera gen echo $TERA_GEN_CMD eval $TERA_GEN_CMD if [ $? -ne 0 ]; then echo "tera gen failed." Fi echo "Teragen passed starting terasort... #run tera sort echo $TERA_SORT_CMD eval $TERA_SORT_CMD if [ $? -ne 0 ]; then echo "tera sort failed." Fi echo "Terasort passed starting teravalidate... #run tera validate echo $TERA_VALIDATE_CMD eval $TERA_VALIDATE_CMD if [ $? -ne 0 ]; then echo "tera validate failed." Fi echo "teragen, terasort, teravalidate passed. echo "Cleaning the data created by tests: $PARENT_DIR"CLEANUP_CMD="su -c '$HADOOP_CMD dfs -rmr -skiptrash $PARENT_DIR' $TEST_USER echo $CLEANUP_CMD eval $CLEANUP_CMD exit 0 Connect to Hadoop Capture Output & Log Run Step 1 and capture exit code Run Step 2 and capture exit code Run Step 3 and capture exit code Cleanup Copyright 6/4/2014 BMC Software, Inc 47

5 Define Design Testing Deploy Develop Maintain Example #!/usr/bin/env bash bin=`dirname "$0"` bin=`cd "$bin"; pwd`. "$bin"/../libexec/hadoop-config.sh #set the hadoop command and the path to the hadoop jar HADOOP_CMD="${HADOOP_PREFIX}/bin/hadoop --config $HADOOP_CONF_DIR #find the hadoop jar HADOOP_JAR=' #find under HADOOP_PREFIX (tar ball install) HADOOP_JAR=`find ${HADOOP_PREFIX} -name 'hadoop--*.jar' head -n1` #if its not found look under /usr/share/hadoop (rpm/deb installs) if [ "$HADOOP_JAR" == '' ]then HADOOP_JAR=`find /usr/share/hadoop -name 'hadoop--*.jar' head -n1` fi #if it is still empty then dont run the tests if [ "$HADOOP_JAR" == '' ]then echo "Did not find hadoop--*.jar under '${HADOOP_PREFIX} or '/usr/share/hadoop'" fi #dir where to store the data on hdfs. The data is relative of the users home dir on hdfs. PARENT_DIR="validate_deploy_`date+%s` TERA_GEN_OUTPUT_DIR="${PARENT_DIR}/tera_gen_data TERA_SORT_OUTPUT_DIR="${PARENT_DIR}/tera_sort_data TERA_VALIDATE_OUTPUT_DIR="${PARENT_DIR}/tera_validate_data Control-M #tera gen cmd TERA_GEN_CMD="su -c '$HADOOP_CMD jar $HADOOP_JAR teragen $TERA_GEN_OUTPUT_DIR' $TEST_USER #tera sort cmd TERA_SORT_CMD="su -c '$HADOOP_CMD jar $HADOOP_JAR terasort $TERA_GEN_OUTPUT_DIR $TERA_SORT_OUTPUT_DIR' $TEST_USER #tera validate cmd TERA_VALIDATE_CMD="su -c '$HADOOP_CMD jar $HADOOP_JAR teravalidate $TERA_SORT_OUTPUT_DIR $TERA_VALIDATE_OUTPUT_DIR' $TEST_USER echo "Starting teragen... #run tera gen echo $TERA_GEN_CMD eval $TERA_GEN_CMD if [ $? -ne 0 ]; then echo "tera gen failed." Fi Environment configuration unnecessary with Control-M. Output & Log Management unnecessary with Control-M. 48 Copyright 6/4/2014 BMC Software, Inc Copyright 6/4/2014 BMC Software, Inc 48

6 BMC Control-M for Hadoop Manage Hadoop batch processing with the same power and ease of your enterprise business processing Faster application implementation Simplify the development of batch workflows with a drag and drop interface that is integrated with Hadoop projects Improve service delivery Detect slowdowns and failures with predictive analytics and intelligent monitoring of workflows Enable rapid business change Connect Hadoop workflows to enterprise processing for an end-to-end view of service Copyright 6/4/2014 BMC Software, Inc 49

7 50 Copyright 6/4/2014 BMC Software, Inc Copyright 6/4/2014 BMC Software, Inc 50

8 Monitoring Job Tracker report Copyright 6/4/2014 BMC Software, Inc 51

9 BMC Control-M Workload Automation Hadoop Application Developers Write programs Build Hadoop jobs Add Pre/Post Jobs Access for the Business Pig Hive MapReduce Sqoop HDFS File Watcher IT Scheduler Copyright 6/4/2014 BMC Software, Inc 52

10 53 Copyright 6/4/2014 BMC Software, Inc Copyright 6/4/2014 BMC Software, Inc 53

11 For Additional Information: Copyright 6/4/2014 BMC Software, Inc 54