Integrating SAS and Elasticsearch: Performing Text Indexing and Search
|
|
- Ashley Roberts
- 6 years ago
- Views:
Transcription
1 Integrating SAS and Elasticsearch: Performing Text Indexing and Search Presented by Edmond Cheng Booz Allen Hamilton SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies.
2 INTEGRATING SAS AND ELASTICSEARCH: PERFORMING TEXT INDEXING AND SEARCH A guide on using Elasticsearch to expand the power of performing textual analysis in SAS products with fast and scalable documents indexing, complex search queries, and rapid visualization QUERYING, SEARCHING, AND STORING SAS can query and search documents by sending HTTP requests via Elasticsearch Search API. Save the JSON-based response in SAS dataset, allowing SAS applications to enrich the results and build analytical models GET Search API SAS AND ELASTIC TEXT SOLUTIONS SAS text specialized products enables text analysis, natural language processing, and statistical model application. Elasticsearch is a popular, fast, and scalable search solution, powering many modern webpages. INDEXING DOCUMENTS TO ELASTICSEARCH Elasticsearch uses standard RESTful APIs and JSON to perform indexing and search operations, mapping, and analytics functions. SAS PROC HTTP and JSON Library handles APIs and contents outputs URI SEARCH REQUEST BODY SEARCH Simple query encoding search parameters in the URL Complex searches supporting JSON Query DSL in message body Indexing API BASE SAS Storing in SAS FIND, VERIFY, SUBSTR, INDEX, TRANWRD, SCAN, COMPARE, SPESID, COMPGED, CONTAINS, LIKE, and etc. REST APIs d JSONs 1 PROC HTTP to request a search 3 MERGE by ordinal values PRODUCTS SAS Text Miner SAS Sentiment Analysis SAS Content Categorization SAS Contextual Analysis SAS Visual Text Analytics PROC HTTP d READING APACHE WEB LOGS TO SAS SAS codes performs extract, transform, load (ETL) operation on NASA web log as INFILE, SAS Perl RegEXP parses common log format to individual fields Web Logs SAS Studio JSON LIB PUT Single Index 1 Defining Index PUT URL and content URL -east1.es.amazonaws.com/nasa/doc/1 2 PROC HTTP sends the INDEX PUT request 1 SAS INFILE to read the log file CONTENT ANALYSIS AND VISUALIZATION Kibana is an interactive analysis tool providing powerful search features and interactive analytics dashboard for performing data exploration and text mining, complementing contextual analysis alongside SAS products. Dashboard {"ipaddr":" ","ide ntity":"-","auth":"","timestamp":"1995/07/01 00:00:01","method":"GET","pat h":"/history/apollo/", "http":"http/1.0","status":"2 00","size":6245} 1.8M HTTP Requests [01/Jul/1995:00:00: ] "GET /history/apollo/ HTTP/1.0" unicomp6.unicomp.net - - [01/Jul/1995:00:00: ] "GET /shuttle/countdown/ HTTP/1.0" [01/Jul/1995:00:00: ] "GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0" burger.letters.com - - [01/Jul/1995:00:00: ] "GET /shuttle/countdown/liftoff.html HTTP/1.0" Assign JSON library to respfile POST Bulk Load 2 RegEXP captures fields 1 Creating the JSON metadata and content file Line 1: {"index" : {"_index" : "nasa", "_type" : "doc", "_id" : " "} Line 2: {"ipaddr":" ","identity":"","auth":"-","timestamp":"1995/07/26 04:58:45", "method":"get","path":"/shuttle/missions/sts70/news/","http":"http/1.0","status":"200", "size":3306} 3 DATA STEP writes to SAS dataset 2 PROC HTTP sends the BULK POST request cheng_edmond@bah.com SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies.
3 Paper Integrating SAS and Elasticsearch: Performing Text Indexing and Search Edmond Cheng, Booz Allen Hamilton ABSTRACT Integrating Elasticsearch document indexing and text search components expands the power of performing textual analysis with SAS solution products. Information technology, digitization, social connection, modern data storage, and big data accelerate unstructured text data production. Understanding the advantage in processing textual data and extracting underlying information provides valuable insights, setting an edge in competition, as seen in e-commerce, internet companies, communication media, marketing, health care, and across many industrial sectors. This paper covers the benefits of architecting and implementing Elasticsearch applications alongside with SAS solutions. The first section presents an overview of Elasticsearch and common use cases. The paper demonstrates indexing SAS datasets into Elasticsearch NoSQL index, writing SAS codes to pass Elasticsearch REST APIs, and storing search query results. The final section demonstrates the use of Elasticsearch Kibana to further complement data visualization and business intelligent reporting capability with SAS analytics. INTRODUCTION SAS TEXT ANALYSIS AND SOLUTIONS SAS as a leading analytics software company, offers a variety of specialized products for performing unstructured textual data processing and analytics. These solutions enable developers and end-users to discover meaningful context, apply statistical model application, and perform text searches to produce relevant information for their use cases. The BASE SAS software includes text manipulation, parsing and, analyze functions for programmers to work with string datatype. BASE SAS functions such as variations of FIND, VERIFY, SUBSTR, INDEX, TRANWRD, SCAN, COMPARE, SPESID, COMPGED, and use of CONTAINS and LIKE with support for wildcards can handle text searches at the record or variable level. The SAS text product line includes: SAS Text Miner, a component of SAS Enterprise Miner, provides advanced text extraction and natural language process features, combining quantitative variables with unstructured text to perform data mining techniques. SAS Sentiment Analysis uses a combination of statistical and linguistic rules to identify sentiment in multiple level of the collection record, category, overall document. SAS Contextual Analysis, often works with SAS Visual Analytics, uses contextual analysis categorizing textual data and building concept models, furthermore integrating results with other SAS products. SAS Visual Text Analytics, which runs on SAS Viya, uses comprehensive solution to identity topic, categorizing text data, building models, and run sentiment analysis. All of these running on difference SAS platforms offer a range of text processing capabilities for customer realize the analytical value of text-based data. ELASTIC STACK The most popular search solution is Elasticsearch developed by the company Elastic. Elasticsearch was developed to be the fastest and easiest to use search engine. If you have browsed Wikipedia, Yelp, or Netflix, then you have run a search powered by Elasticsearch. The search engine is based on Lucene, which is a free and open-source library for performing full text indexing and searching capability. Elasticsearch further provides distributed storage, a JSON-based common interface, RESTful APIs, and high accessibility for client applications. The official clients are available in Java,.NET, PHP, Python, Ruby, Groovy, and many other languages. The product opensource development continues to releasing new search attributes, adding new analytical functions, and improving enterprise features supporting customers with wide variety of use cases based on Elastic Stack. 1
4 Application Developers and Data Engineers are finding new use of Elasticsearch for storing documents and records with its build-in NoSQL like database. Other use cases leverage Elasticsearch for running advanced analytics on textual data, outperforming enterprise-grade database and Hadoop systems. The Elastics Stack is a combination of three tools Elasticsearch, Logstash, and Kibana. Logstash serves as a data processing pipeline that ingests data from multiple data sources, performs transformations, send the data to Elasticsearch. Kibana is a visualization tool to work with documents stored in Elasticsearch. The rest of this paper demonstrates integrating SAS with Elastic Stack using document indexing and search results to perform text analysis. NASA WEB TRAFFIC LOG The data is a public release of NASA Kennedy Space Center web server logs collected in 1995 July containing approximately 1.8 million logs. The data is publicly available in Total size of the uncompressed file is 205.2MB. The log file is in Apache common log format: {host} {identify} {authentication} [{timestamp}] {request} {reply_code} {size} Example: [01/Jul/1995:00:00: ] "GET /history/apollo/ HTTP/1.0" unicomp6.unicomp.net - - [01/Jul/1995:00:00: ] "GET /shuttle/countdown/ HTTP/1.0" [01/Jul/1995:00:00: ] "GET /shuttle/missions/sts-73/missionsts-73.html HTTP/1.0" burger.letters.com - - [01/Jul/1995:00:00: ] "GET /shuttle/countdown/liftoff.html HTTP/1.0" [01/Jul/1995:00:00: ] "GET /shuttle/missions/sts-73/sts-73- patch-small.gif HTTP/1.0" SETTING UP THE ELASTICSEARCH CLUSTER The Elasticsearch cluster is provisioned in Amazon AWS using the AWS Elasticsearch service. The ease of quickly deploying a healthy 3-node ES cluster with security enabled and high performance save the headaches in manually installing and configuring your own cluster for prototyping use cases. The AWS service offers ES 5.x/6.x versions, various EC2 instance and storage options, and setting access policy. Once the AWS Elasticsearch cluster is active, the service creates the Elasticsearch and Kibana endpoints. SAS will be to use PROC HTTP to communicate Elasticsearch RESTful API for passing PUT, POST, and GET HTTP requests. 2
5 Note: the paper only shows the essential codes illustrating the syntax and technique to accomplish the task. See the Appendix for the complete sample SAS program READING LOG FILE TO SAS For data processing, SAS codes are programmed to perform light extract, transform, load (ETL) operation on reading the NASA web log file and writing to a SAS dataset. The NASA web log file can be handled by using INFILE function and SAS Perl Regular Expression. First, define the regular expression to match the common log format. Within the REQUEST message, the rx expression is extended to further capture the METHOD, PATH, and HTTP fields. The PRXPARSE function compiles the RX regex expression. Use PRXNEXT, PRXMATCH, and PRXPOSN to parses each line of the record to 9 capture groups. This example stores all incoming fields as string datatype. A different DATA STEP later has a separate routine to convert all the fields to JSON notation, it is optional or rather simpler storing numeric datatype as string for ease of data manipulation. The timestamp variable needs to be put in a datetime format matching to an Elasticsearch built-in date format. Elasticsearch is preconfigured with a set of recognizable date formats and able to parse date formatted as string into supported datatime values. With properly define date format, Elasticsearch can perform additional time-based analysis and aggregation. Once the DATA STEP completes reading the log file, run a PROC PRINT of the SAS dataset to verify the capture groups are properly stored and formatted. Also run PROC JSON to print the outputs in JSON format verifying formatting issues before moving to the indexing section. 3
6 Output from PROC PRINT: Output from PROC JSON: INDEXING DOCUMENTS USING PUT AND BULK LOAD Elasticsearch uses standard RESTful APIs and JSON based data exchange to perform all indexing and search operations, plus other analytics functions, mapping, and cluster administration. Elasticsearch provides excellent documentation on its API library, including examples which can run directly in client application, command line, and Kibana. You can access the APIs using standard HTTP, or any client technology, such as Java, Python,.NET, C, C#, Perl, Ruby, and other languages. Recent release of SAS 9.4 brought number of improvements to PROC HTTP and JSON Library, increasing the flexibility in handling custom header and application contents. The learning curve to integrate Elasticsearch to SAS development is mostly translating SAS DATA STEP to prepare the API request URL and content body, using PROC HTTP to call the Elasticsearch API URL endpoint defined for the cluster. To get familiar with sending messages to the Elasticsearch cluster for verification, define the API endpoint and index information as macro parameters, then use PROD HTTP to send the API call. 4
7 If the messages successfully reach the Elasticsearch server, the service will response with a response message showing the Elasticsearch cluster information. The header message shows HTTP 200 OK status The response message shows information about the cluster INDEXING METHOD 1: PUT INDEX API Elasticsearch provides two API methods for indexing documents. The first option is the INDEX API that involves sending a HTTP PUT request to insert or update a single document. The URL specification includes the cluster endpoint, index, type, and id number. The body of the message is JSON based. One important and popular Elasticsearch feature is dynamic mapping, without having to define an initial mapping nor modifying schema for mapping changes when ingesting documents. Therefore, if an initial mapping is not found for the index, a new mapping will be dynamically generated. Reading from the nasa_log.sas7bdat dataset, use a DO-LOOP converts the character and number variables to individual key-pair store conforming to JSON format. Notice the code has a slight variation for handling numeric datatype because quoting is not needed around the value. Writing the URL is simply defining macro parameters for the Elasticsearch endpoint &ES, index name &INDEX, and type &TYPE concatenated with the ID value, taken from _n_ reference. This example produces URL and CONTENT as follows: URL = 1.es.amazonaws.com/nasa/doc/1 CONTENT = {"ipaddr":" ","identity":"-","auth":"-","timestamp": "1995/07/01 00:00:01","method":"GET","path":"/history/apollo/", "http":"http/1.0","status":"200","size":6245} Next, store the URL and CONTENT as a series of macro variable sequence 5
8 Lastly, using PROC HTTP to pass the INDEX PUT requests with the macroized &&URL&i and &&CONTENT&I to the Elasticsearch cluster. Macro parameterized DO-LOOP is not shown. INDEXING METHOD 2: BULK LOAD API Since the HTTP request can insert only one document at a time, the PUT INDEX API is not suitable for indexing large volume of records. Indexing 1.8 million NASA log messages would take more than 20 hours. The second method is the Bulk Load API, it performs multiple index operations in a single HTTP POST API call. The Bulk Load API can insert thousands of records, within the HTTP transfer limit per the Elasticsearch server configuration. For example, AWS Elasticsearch service places a 10MB or 100MB HTTP transfer limit per single request, depending on the instance type. Regardless, the Bulk Load API is what you want SAS to interface when indexing to Elasticsearch. The Bulk data format specification is a JSON-based file containing two lines for each record, a metadata and content entry, separated by a newline. The first line contains the metadata information including the index, type, and ID. The second line contains the message body in JSON object, which is also identical to the CONTENT format used in PUT INDEX API. Create a new SAS dataset based on the previous PUT Index dataset. For each record, output a new record for the metadata key-value pairs and output another record for the content. This example produces Line-1 and Line-2 for each record as follow: Line 1: {"index" : {"_index" : "nasa", "_type" : "doc", "_id" : " "} Line 2: {"ipaddr":" ","identity":"-","auth":"-","timestamp": "1995/07/26 04:58:45","method":"GET","path":"/shuttle/missions/sts- 70/news/","http":"HTTP/1.0","status":"200","size":3306} Prepare a JSON-based input file containing all the documents to be ingested by Bulk Load API 6
9 Contents from file ingest: Lastly, use PROC HTTP to pass the Bulk Load Post request, sending the input file to the Elasticsearch cluster While not shown, the HEADEROUT=HDRSFILE and OUT=RESPFILE captures the Elasticsearch response. If the index is successful, the header file will show STATUS 200, and the response file will show all the indexed documents as JSON objects. QUERYING, SEARCHING, AND STORING AS SAS DATASET If you follow the Elasticsearch release cycle, the product opensource development moves at a rapid pace. Every Elasticsearch release offers new features and improvements. This paper demonstrates only few SEARCH API examples in context of integrating with SAS. In the ES documentation, select the version matching the working ES cluster and examine the Search APIs. The first part demonstrates how SAS sends GET HTTP request calling the Search API to query or search document. The second part shows how SAS stores the JSON-based response as SAS dataset. By storing the Elasticsearch results in SAS, SAS can consume the results and further build analysis models for insightful reporting. SEARCH METHOD 1: URI SEARCH Elasticsearch provides two Search APIs. The URI Search or query lite API sends a HTTP GET request with the required parameters encoded in the URL. URI Search is a simple way to query for results without passing message body. However, the number of search options are limited in this mode. Example 1: Query document ID and print pretty SAS Code Result 7
10 Example 2: Search thru the web logs for terms 'apollo or discovery' and STATUS equals to note the URL encoding SAS Code Partial Result SEARCH METHOD 2: REQUEST BODY SEARCH The Request Body Search method sends a JSON body as part of the request search. The JSON body is a Query DSL (Domain Specific Language) based on a structured format to define the query. The query uses reserved words to perform operations, these terms include match, term, range, and bool. The Request Body Search handles the most extensive and complex searches, including aggregation, multi fields, and fuzzy search. See the Elasticsearch documentation for full functionality. Example 3: Search thru the web logs for the top 100 relevant results with the terms matching apollo, status code equals 200, byte size greater than 100kb and sort in descend order by timestamp. SAS Code Partial Result STORING ELASTICSEARCH RESULTS IN SAS The Elasticsearch speed and extensive text search algorithm can enrich text analysis capabilities in SAS. Elasticsearch executes complex searches across multiple indices and takes advantage of distributed 8
11 shards and replicas, returning results faster than SAS functions can produce. Elasticsearch allows fuzzy searches and relevance scoring return closest matching results taking consideration of language processing engine. Elasticsearch could replace time intensity regex running in SAS DATA STEP code. SAS Stored Procedure and ODS interfacing with Elasticsearch PUT and GET API can be added to Visual Analytics extending customized keyword search capability when viewing and search reports. SAS 9.4M4 released the JSON Library allowing data access to JSON document as a SAS library reference This method allows storing JSON output returned from Elasticsearch as SAS dataset. Assigning the JSON Library reads the JSON file as a SAS dataset. An optional JSON map can be provided to define the JSON output schema. If none is specified, the SAS JSON library will automatically generate a JSON map based on the data. Use PROC HTTP to call the URI Search API to query for first 100 results Assign a JSON library referencing the SEARCH response file, the library assignment reads the JSON file and auto-generates a hierarchy of the table structure. To examine the library contents, run PROC DATASET and PROC CONTENTS to explore the table and column references. In this example, the HITS_HITS table contains the original document ID. The HITS SOURCE table contains the message content. Therefore to produce a SAS DATASET with doc id #, the two datasets needs to be merged. Lastly, verify the new SAS dataset is properly storing the outputs Outputs from PROC PRINT: TEXT ANALYSIS IN VISUALIZATION Elastic Stack provides Kibana as an interactive development tool and visualization web console operating on top of Elasticsearch. Developers can use its Dev Tools to perform operations on the cluster and indexes, 9
12 instead of passing JSON messages thru curl commands. Users can access Dashboard created with pre-defined filters and shared visualizations. The Dashboard supports extensive Elasticsearch search operations and Lucene search syntax, plus interactive filters on all visualizations in a dashboard. Visualizations from multiple indices can be displayed in given dashboard allowing searches across different collections. Kibana is a good complementary tool for performing text mining and terms exploration when building contextual models using SAS products. The Kibana Discover tool allows user entering a search query matching over 1.8 million log entries with IP address ending in.edu or.gov. The application took less than a second to paginate the first batch of results. In some way, searching in Discover can complement SAS users who work or without SAS text mining solutions to discover keywords when building synonyms or topics for contextual analysis. Kibana Dashboard impressively displays multiple interactive charting on the NASA web logs loaded from SAS. The time series chart plots the daily aggregated server web requests handled by the server. The table shows the hostname of the top ten requestors. The bar charts show the total packet size returned to the top ten requestors, and the pie chart indicates the GET method leading all HTTP requests to the server. While functionality and customization are limited comparing to SAS Visual Analytics, Kibana offers powerful search analytics and interactive dashboard for data exploration and reporting. 10
13 CONCLUSION The significant growth and difficulty of processing unstructured textual data analysis demands fast, flexible, and efficient approach in discovering relevant information. SAS products offers number of solutions with unique approaches in supporting text data mining, topic identification, concept building, and combining quantitative variables with textual context for analysis. Elasticsearch is a powerful tool for indexing and searching over large volume of documents returning results in milliseconds, plus a robust RESTful API library and JSON data common interface. Integrating SAS solutions with other search-based applications like Elastic Stack further complements the exploration, enrichment, and reporting key information in textual data. REFERENCES DeVenezia, Richard A. Parsing Apache log files. Devenezia. March 1, Elastic. Elasticsearch Reference 5.5. Docs. March 1, SAS Institute, Inc. HTTP Procedure. Base SAS 9.4 Procedure Guide, Seventh Edition. March 1, etversion=9.4&locale=en. The Internet Traffic Archive. NASA-HTTP. Contrib. March 1, Windham, Mathew Introduction to Regular Expressions. 1 st ed. SAS Institute. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Edmond Cheng Booz Allen Hamilton cheng_edmond@bah.com SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 11
14 APPENDIX /******************************************* * PROGRAM: SAS_Elasticsearch.sas * DESCRIPTION: Demonstration of expanding Base SAS textual analysis with Elasticsearch func: * - Indexing high volume of documents from SAS to ES * - Querying ES indexes and storing results in SAS dataset * - Sending SAS dataset to Kibana for data visualization and text analysis * INPUTS NASA http log in Common Log Format from The Internet Traffic Archive, * * OUTPUTS SAS dataset * Elasticsearch index * PROGRAMMER: Edmond Cheng * DATE: * NOTES: *******************************************/ /***** OPTIONS *****/ options nomprint compress=yes reuse=yes; proc printto log='/folders/myfolders/saslogs/saslog.log' print='/folders/myfolders/saslists/saslist.lst' new; /***** PARAMETERS *****/ %let nasafile = '/folders/myfolders/rawdata/nasa_access_log_jul95'; %let nasajson = '/folders/myfolders/outputs/nasa_log.json'; %let es 1.es.amazonaws.com; %let index %let type = nasa; = doc; = /***** ENVIRONMENT *****/ libname saslib '/folders/myfolders/sasdata'; /* SAS library */ filename hdrsfile '/folders/myfolders/outputs/hdrs.txt'; /* fileref for response headers */ filename respfile '/folders/myfolders/outputs/resp.json'; /* fileref for return output */ filename ingest '/folders/myfolders/outputs/ingest.json'; /* fileref for ingest file */ /***** FORMATS *****/ proc format; picture dt other='%0y/%0m/%0d %0H:%0M:%0S' (datatype=datetime); /***** MACROS *****/ %macro response_outputs(send_to_log); %put 'PRINTING Response Header'; data _null_; infile hdrsfile; input; %if &send_to_log=yes %then put _infile_;; %put 'PRINTING HTTP Output'; data _null_; infile respfile; input; %if &send_to_log=yes %then put _infile_;; %mend response_outputs; 12
15 /***** READING COMMON LOG FILE *****/ /* Reference code from Richard A. DeVenezia, */ /* Read in the raw nasa http log file and parse out the elements */ data nasa_log; retain rx; infile "&nasafile" lrecl=32767 end=end; if _N_ = 1 then rx = prxparse('/^(\s+) (\S+) (\S+) \[(.+)\] \"(\S+) (\S+)\s?(.*)\" (\S+) (\S+)/') ; * process each line of log file; do i = 1 until (end); input; /* Define pattern matching */ start=1; stop = length (_infile_); call prxnext (rx,start,stop,_infile_,matchat,matchlength); /* Extract */ if prxmatch (rx, _infile_) then do; id = strip(put(_n_,best12.)); ipaddr = prxposn (rx, 1, _infile_); identity = prxposn (rx, 2, _infile_); auth = prxposn (rx, 3, _infile_); timestamp = prxposn (rx, 4, _infile_); method = prxposn (rx, 5, _infile_); path = prxposn (rx, 6, _infile_); http = prxposn (rx, 7, _infile_); status = prxposn (rx, 8, _infile_); size = prxposn (rx, 9, _infile_); /* Transform */ timestamp = strip(put(input(compress(timestamp,'/'),datetime.),dt.)); /* Load */ output; end; end; else put _n_ 'data line in Apache log file is not in common format'; drop i rx start stop matchat matchlength; /* Verify outputs in 1) proper SAS data format and 2) JSON file */ proc print data=nasa_log (obs=10); proc json out="&nasajson" nosastags pretty; export nasa_log (obs=10); /***** VERIFICATION *****/ /* Verifying connectivity to Elasticsearch cluster */ proc http url="&es" 13
16 out=respfile headerout=hdrsfile; %response_outputs(send_to_log=yes); /* Print a list of existing indexes */ proc http url="&es/_cat/indices?v" out=respfile headerout=hdrsfile; %response_outputs(send_to_log=yes); /***** INDEXING *****/ /***** INDEXING: PUT INDEX API */ /* Using single PUT _index API to index one document *****/ /* Step 1: Prepare URL and Content to be passed into ES in the SAS dataset URL is the http with the ES address, index name, type, and record ID Content is the message body in JSON object */ %let charvar = ipaddr identity auth timestamp method path http status ; %let numvar = size; data es_index_put; length url content $500; set nasa_log (obs=max); array charvar{*} &charvar ; array numvar{*} &numvar ; array allvar{*} &charvar &numvar; do i = 1 to dim(charvar) ; charvar{i}=cats('"',vname(charvar{i}),'":"',charvar{i},'"'); end ; do i = 1 to dim(numvar) ; if numvar{i} = '-' then numvar{i} = '0'; numvar{i}=cats('"',vname(numvar{i}),'":',numvar{i}); end ; * URL and Content ; url = cats("&es./&index./&type./",id); content = cats("{",catx(",", of allvar{*}),"}"); /* Step 2: Generate the sequence of url and content string as macro variables */ proc sql noprint; select url, content into :url1-:url , :content1-:content from es_index_put; quit; %put The first and last definitions are:; %put #1: url=&url1; %put #1: content=&content1; %put #&sqlobs: url=&url1; %put #&sqlobs: content=&content1; /* Step 3: Using Proc HTTP to pass the ES _index PUT API with the URL/Content */ %macro put_es; %do i = 1 %to &sqlobs; proc http url="&&url&i" /* SAS generated ES PUT Index URL */ method="put" in= %unquote(%str(%'&&content&i%')) /* SAS generated record json */ out=respfile 14
17 headerout=hdrsfile; %response_outputs(send_to_log=no); %end; %mend put_es; %put_es; /***** INDEXING: BULK LOAD API */ /* Using bulk index API to put batches of docs *****/ /* Step 1: The bulk load API requires a metadata and body entry for each record, separated be a newline Bulk Content line 1 contains the index name, type, and id in JSON object Bulk Content line 2 contains the message body in JSON object, also identical with index API */ %macro es_index_bulk(firstnobs=,nobs=); data es_index_bulk; length bulk_content $500; set es_index_put (firstobs=&firstnobs obs=&nobs); bulk_content = cats('{"index" : {"_index" : "',"&index",'", "_type" : "',"&type", '", "_id" : "',id,'"}'); output; bulk_content = content; output; keep bulk_content; /* Step 2: Prepare an input file to be ingested by bulk api */ data _null_; set es_index_bulk; file ingest; put bulk_content; /* Step 3: Using SAS Proc HTTP to define the ingest file in ES _bulk POST API */ proc http url="&es/_bulk" /* ES POST Bulk API */ in=ingest /* SAS generated bulk records json */ out=respfile headerout=hdrsfile; headers "Content-Type"="application/x-ndjson"; %response_outputs(send_to_log=no); %mend es_index_bulk; /* Run in batches of 250,000 records conforming to server bandwidth */ %es_index_bulk(firstnobs=1,nobs=250000); %es_index_bulk(firstnobs=250001,nobs=500000); %es_index_bulk(firstnobs=500001,nobs=750000); %es_index_bulk(firstnobs=750001,nobs= ); %es_index_bulk(firstnobs= ,nobs= ); %es_index_bulk(firstnobs= ,nobs= ); %es_index_bulk(firstnobs= ,nobs= ); %es_index_bulk(firstnobs= ,nobs= ); /***** QUERY, SEARCH, AND TEXT ANALYSIS *****/ /* Step 1 Using GET query lite API to query docs */ /* Query document ID */ proc http url="&es/&index/&type/13250?pretty" headerout=hdrsfile out=respfile; 15
18 %response_outputs(send_to_log=yes); /* Search for http log with term 'apollo or discovery' and status '200' OK - note the URL encoding */ proc http url="&es/&index/&type/_search?q=%2bpath%3a(apollo+discovery)+%2bstatus%3a200&pretty" headerout=hdrsfile out=respfile; %response_outputs(send_to_log=yes); /* Step 2 Using GET request body search allowing more complex and multi fields search Search for top 100 best matching http logs with term 'apollo, status 200 OK code, size greater than 100kb and sort in descend order by timestamp */ proc http url="&es/&index/&type/_search" in='{ "query": { "bool":{ "must":{"term":{"path":"mission"}}, "filter":{"term":{"status":"200"}}, "filter":{"range":{"size":{"gte":100000}}} } }, "sort" : {"timestamp" : {"order" : "desc"}}, "size":100 } ' headerout=hdrsfile out=respfile; %response_outputs(send_to_log=yes); /* Step 3 Storing the results in SAS dataset */ /* Set a pagination size of 1,000 documents */ %let size=100; proc http url="&es/&index/&type/_search?size=&size" /* ES Query lite API */ out=respfile results to a response file */ headerout=hdrsfile; %response_outputs(send_to_log=no); /* Output the /* Set up a JSON library referenceing the response file output from ES */ /* The JSON file is represented by auto generated hierarchy of tables, hits Use the proc dataset and content to explore the appropiate table and columns needed */ libname es JSON fileref=respfile; proc datasets lib=es; quit; /* In this example, The hits_hits table contains the document id # The hits source table contains the message content To produce a complete record with the doc id #, the two datasets are merged by the SAS generated ordinal_hits field */ data es_results; merge es.hits_hits (in=a drop=ordinal_hits rename=(ordinal_hits1=ordinal_hits)) es.hits source (in=b); by ordinal_hits; if a and b; /* Verify outputs showing the first 10 matched records sorted by relevance score */ proc print data=es_results (obs=10); 16
Goal of this document: A simple yet effective
INTRODUCTION TO ELK STACK Goal of this document: A simple yet effective document for folks who want to learn basics of ELK (Elasticsearch, Logstash and Kibana) without any prior knowledge. Introduction:
Amazon Elasticsearch Service
Amazon Elasticsearch Service Fully managed, reliable, and scalable Elasticsearch service. Have Your Frontend & Monitor It Too Scalable Log Analytics Inside a VPC Lab Instructions Contents Lab Overview...
Table 1 The Elastic Stack use cases Use case Industry or vertical market Operational log analytics: Gain real-time operational insight, reduce Mean Ti
Solution Overview Cisco UCS Integrated Infrastructure for Big Data with the Elastic Stack Cisco and Elastic deliver a powerful, scalable, and programmable IT operations and security analytics platform
Guide Users along Information Pathways and Surf through the Data
Guide Users along Information Pathways and Surf through the Data Stephen Overton, Overton Technologies, LLC, Raleigh, NC ABSTRACT Business information can be consumed many ways using the SAS Enterprise
by Cisco Intercloud Fabric and the Cisco
Expand Your Data Search and Analysis Capability Across a Hybrid Cloud Solution Brief June 2015 Highlights Extend Your Data Center and Cloud Build a hybrid cloud from your IT resources and public and providerhosted
Search Engines and Time Series Databases
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Search Engines and Time Series Databases Corso di Sistemi e Architetture per Big Data A.A. 2017/18
Data Mining with Elastic
2017 IJSRST Volume 3 Issue 3 Print ISSN: 2395-6011 Online ISSN: 2395-602X Themed Section: Science and Technology Data Mining with Elastic Mani Nandhini Sri, Mani Nivedhini, Dr. A. Balamurugan Sri Krishna
Improving the prediction of next page request by a web user using Page Rank algorithm
Improving the prediction of next page request by a web user using Page Rank algorithm Claudia Elena Dinucă, Dumitru Ciobanu Faculty of Economics and Business Administration Cybernetics and statistics University
Shine a Light on Dark Data with Vertica Flex Tables
White Paper Analytics and Big Data Shine a Light on Dark Data with Vertica Flex Tables Hidden within the dark recesses of your enterprise lurks dark data, information that exists but is forgotten, unused,
ADVANCED DATABASES CIS 6930 Dr. Markus Schneider. Group 5 Ajantha Ramineni, Sahil Tiwari, Rishabh Jain, Shivang Gupta
ADVANCED DATABASES CIS 6930 Dr. Markus Schneider Group 5 Ajantha Ramineni, Sahil Tiwari, Rishabh Jain, Shivang Gupta WHAT IS ELASTIC SEARCH? Elastic Search Elasticsearch is a search engine based on Lucene.
Percona Live September 21-23, 2015 Mövenpick Hotel Amsterdam
Percona Live 2015 September 21-23, 2015 Mövenpick Hotel Amsterdam MongoDB, Elastic, and Hadoop: The What, When, and How Kimberly Wilkins Principal Engineer/Database Denizen ObjectRocket/Rackspace kimberly@objectrocket.com
Improving Drupal search experience with Apache Solr and Elasticsearch
Improving Drupal search experience with Apache Solr and Elasticsearch Milos Pumpalovic Web Front-end Developer Gene Mohr Web Back-end Developer About Us Milos Pumpalovic Front End Developer Drupal theming
Seamless Dynamic Web (and Smart Device!) Reporting with SAS D.J. Penix, Pinnacle Solutions, Indianapolis, IN
Paper RIV05 Seamless Dynamic Web (and Smart Device!) Reporting with SAS D.J. Penix, Pinnacle Solutions, Indianapolis, IN ABSTRACT The SAS Business Intelligence platform provides a wide variety of reporting
Introducing SAS Model Manager 15.1 for SAS Viya
ABSTRACT Paper SAS2284-2018 Introducing SAS Model Manager 15.1 for SAS Viya Glenn Clingroth, Robert Chu, Steve Sparano, David Duling SAS Institute Inc. SAS Model Manager has been a popular product since
ABSTRACT: INTRODUCTION: WEB CRAWLER OVERVIEW: METHOD 1: WEB CRAWLER IN SAS DATA STEP CODE. Paper CC-17
Paper CC-17 Your Friendly Neighborhood Web Crawler: A Guide to Crawling the Web with SAS Jake Bartlett, Alicia Bieringer, and James Cox PhD, SAS Institute Inc., Cary, NC ABSTRACT: The World Wide Web has
SAS Viya : The Beauty of REST in Action
ABSTRACT Paper 2712-2018 SAS Viya : The Beauty of REST in Action Sean Ankenbruck and Grace Heyne Lybrand, Zencos Consulting LLC The introduction of SAS Viya opened the SAS brand and its industry leading
Prototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.org Pete Skomoroch Research Scientist at LinkedIn Consultant at Data Wrangling @peteskomoroch 09/29/09 1 Talk Outline TrendingTopics Overview Wikipedia Page
Ninja Level Infrastructure Monitoring. Defensive Approach to Security Monitoring and Automation
Ninja Level Infrastructure Monitoring Defensive Approach to Security Monitoring and Automation 1 DEFCON 24 06 th August 2016, Saturday 10:00-14:00 Madhu Akula & Riyaz Walikar Appsecco.com 2 About Automation
SAS IT Resource Management 3.8: Reporting Guide
SAS IT Resource Management 3.8: Reporting Guide SAS Documentation The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2017. SAS IT Resource Management 3.8: Reporting Guide.
Streaming ETL of High-Velocity Big Data Using SAS Event Stream Processing and SAS Viya
SAS 1679-2018 Streaming ETL of High-Velocity Big Data Using SAS Event Stream Processing and SAS Viya ABSTRACT Joydeep Bhattacharya and Manish Jhunjhunwala, SAS Institute Inc. A typical ETL happens once
Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015
Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL May 2015 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document
About the Tutorial. Audience. Prerequisites. Copyright and Disclaimer. Logstash
About the Tutorial is an open-source, centralized, events and logging manager. It is a part of the ELK (ElasticSearch,, Kibana) stack. In this tutorial, we will understand the basics of, its features,
Creating a Recommender System. An Elasticsearch & Apache Spark approach
Creating a Recommender System An Elasticsearch & Apache Spark approach My Profile SKILLS Álvaro Santos Andrés Big Data & Analytics Solution Architect in Ericsson with more than 12 years of experience focused
Semantic Web Company. PoolParty - Server. PoolParty - Technical White Paper.
Semantic Web Company PoolParty - Server PoolParty - Technical White Paper http://www.poolparty.biz Table of Contents Introduction... 3 PoolParty Technical Overview... 3 PoolParty Components Overview...
ANOMALY DETECTION ON MACHINE LOG
ANOMALY DETECTION ON MACHINE LOG Data Mining Prof. Sunnie S Chung Ankur Pandit 2619650 Raw Data: NASA HTTP access logs It contain two month's of all HTTP requests to the NASA Kennedy Space Center WWW server
OASUS Spring 2014 Questions and Answers
OASUS Spring 2014 Questions and Answers The following answers are provided to the benefit of the OASUS Users Group and are not meant to replace SAS Technical Support. Also, the Enterprise Guide project
Log Analytics with Amazon Elasticsearch Service. Christoph Schmitter
Log Analytics with Amazon Elasticsearch Service Christoph Schmitter (csc@amazon.de) What we'll cover Understanding Elasticsearch capabilities Elasticsearch, the technology Aggregations; ad-hoc analysis
Base and Advance SAS
Base and Advance SAS BASE SAS INTRODUCTION An Overview of the SAS System SAS Tasks Output produced by the SAS System SAS Tools (SAS Program - Data step and Proc step) A sample SAS program Exploring SAS
How to Route Internet Traffic between A Mobile Application and IoT Device?
Whitepaper How to Route Internet Traffic between A Mobile Application and IoT Device? Website: www.mobodexter.com www.paasmer.co 1 Table of Contents 1. Introduction 3 2. Approach: 1 Uses AWS IoT Setup
Search and Time Series Databases
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Search and Time Series Databases Corso di Sistemi e Architetture per Big Data A.A. 2016/17 Valeria
Homework 9: Stock Search Android App with Facebook Post A Mobile Phone Exercise
Homework 9: Stock Search Android App with Facebook Post A Mobile Phone Exercise 1. Objectives Ø Become familiar with Android Studio, Android App development and Facebook SDK for Android. Ø Build a good-looking
Integrate MATLAB Analytics into Enterprise Applications
Integrate Analytics into Enterprise Applications Dr. Roland Michaely 2015 The MathWorks, Inc. 1 Data Analytics Workflow Access and Explore Data Preprocess Data Develop Predictive Models Integrate Analytics
Microservices log gathering, processing and storing
Microservices log gathering, processing and storing Siim-Toomas Marran Univeristy of Tartu J.Liivi 2 Tartu, Estonia siimtoom@ut.ee ABSTRACT The aim of this work is to investigate and implement one of the
Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk
Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Raanan Dagan and Rohit Pujari September 25, 2017 Washington, DC Forward-Looking Statements During the course of this presentation, we may
Give m Their Way They ll Love it! Sarita Prasad Bedge, Family Care Inc., Portland, Oregon
Give m Their Way They ll Love it! Sarita Prasad Bedge, Family Care Inc., Portland, Oregon ABSTRACT As many of us know, many users prefer their SAS data in MS Excel spreadsheet. And many of us also may
Enterprise Data Catalog for Microsoft Azure Tutorial
Enterprise Data Catalog for Microsoft Azure Tutorial VERSION 10.2 JANUARY 2018 Page 1 of 45 Contents Tutorial Objectives... 4 Enterprise Data Catalog Overview... 5 Overview... 5 Objectives... 5 Enterprise
INTRODUCTION TO SAS HOW SAS WORKS READING RAW DATA INTO SAS
TO SAS NEED FOR SAS WHO USES SAS WHAT IS SAS? OVERVIEW OF BASE SAS SOFTWARE DATA MANAGEMENT FACILITY STRUCTURE OF SAS DATASET SAS PROGRAM PROGRAMMING LANGUAGE ELEMENTS OF THE SAS LANGUAGE RULES FOR SAS
Amazon Search Services. Christoph Schmitter
Amazon Search Services Christoph Schmitter csc@amazon.de What we'll cover Overview of Amazon Search Services Understand the difference between Cloudsearch and Amazon ElasticSearch Service Q&A Amazon Search
Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch
Nick Pentreath Nov / 14 / 16 Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch About @MLnick Principal Engineer, IBM Apache Spark PMC Focused on machine learning
Hawaii Energy and Environmental Technologies (HEET) Initiative
Hawaii Energy and Environmental Technologies (HEET) Initiative Office of Naval Research Grant Award Number N0014-11-1-0391 Task 8. ENERGY-NEUTRAL ENERGY TEST PLATFORMS 8.3 Advanced Database Research, Development
Build your own IDM Audit Dashboard
Build your own IDM Audit Dashboard Open Horizons Magazine for OH Summit Budapest 2014 Special Edition Q2, 2014 by Andreas Fuhrmann, SKyPRO AG, Switzerland The NetIQ Identity Manager is a very powerful
Overview. : Cloudera Data Analyst Training. Course Outline :: Cloudera Data Analyst Training::
Module Title Duration : Cloudera Data Analyst Training : 4 days Overview Take your knowledge to the next level Cloudera University s four-day data analyst training course will teach you to apply traditional
Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect
Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect Igor Roiter Big Data Cloud Solution Architect Working as a Data Specialist for the last 11 years 9 of them as a Consultant specializing
Elasticsearch. Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec
Elasticsearch Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec What about Elasticsearch the Company?! Support 100s of Companies in Production environments Training
SAS System Powers Web Measurement Solution at U S WEST
SAS System Powers Web Measurement Solution at U S WEST Bob Romero, U S WEST Communications, Technical Expert - SAS and Data Analysis Dale Hamilton, U S WEST Communications, Capacity Provisioning Process
Processing Unstructured Data. Dinesh Priyankara Founder/Principal Architect dinesql Pvt Ltd.
Processing Unstructured Data Dinesh Priyankara Founder/Principal Architect dinesql Pvt Ltd. http://dinesql.com / Dinesh Priyankara @dinesh_priya Founder/Principal Architect dinesql Pvt Ltd. Microsoft Most
An Oracle White Paper October Oracle Social Cloud Platform Text Analytics
An Oracle White Paper October 2012 Oracle Social Cloud Platform Text Analytics Executive Overview Oracle s social cloud text analytics platform is able to process unstructured text-based conversations
Using the MySQL Document Store
Using the MySQL Document Store Alfredo Kojima, Sr. Software Dev. Manager, MySQL Mike Zinner, Sr. Software Dev. Director, MySQL Safe Harbor Statement The following is intended to outline our general product
SAS Job Monitor 2.2. About SAS Job Monitor. Overview. SAS Job Monitor for SAS Data Integration Studio
SAS Job Monitor 2.2 About SAS Job Monitor Overview SAS Job Monitor is a component of SAS Environment Manager that integrates information from SAS Data Integration Studio, DataFlux Data Management Server,
Blended Learning Outline: Cloudera Data Analyst Training (171219a)
Blended Learning Outline: Cloudera Data Analyst Training (171219a) Cloudera Univeristy s data analyst training course will teach you to apply traditional data analytics and business intelligence skills
SAS ENTERPRISE GUIDE USER INTERFACE
Paper 294-2008 What s New in the 4.2 releases of SAS Enterprise Guide and the SAS Add-In for Microsoft Office I-kong Fu, Lina Clover, and Anand Chitale, SAS Institute Inc., Cary, NC ABSTRACT SAS Enterprise
Compute Service: A RESTful Approach to the SAS Programming Environment
Paper SAS2083-2018 Compute Service: A RESTful Approach to the SAS Programming Environment Jason Spruill and Joseph Henry, SAS Institute Inc., Cary, NC ABSTRACT SAS Viya provides an open architecture that
@Pentaho #BigDataWebSeries
Enterprise Data Warehouse Optimization with Hadoop Big Data @Pentaho #BigDataWebSeries Your Hosts Today Dave Henry SVP Enterprise Solutions Davy Nys VP EMEA & APAC 2 Source/copyright: The Human Face of
SAS 9 Programming Enhancements Marje Fecht, Prowerk Consulting Ltd Mississauga, Ontario, Canada
SAS 9 Programming Enhancements Marje Fecht, Prowerk Consulting Ltd Mississauga, Ontario, Canada ABSTRACT Performance improvements are the well-publicized enhancement to SAS 9, but what else has changed
FAQs. Business (CIP 2.2) AWS Market Place Troubleshooting and FAQ Guide
FAQs 1. What is the browser compatibility for logging into the TCS Connected Intelligence Data Lake for Business Portal? Please check whether you are using Mozilla Firefox 18 or above and Google Chrome
ForeScout CounterACT. Configuration Guide. Version 3.4
ForeScout CounterACT Open Integration Module: Data Exchange Version 3.4 Table of Contents About the Data Exchange Module... 4 About Support for Dual Stack Environments... 4 Requirements... 4 CounterACT
Powering Knowledge Discovery. Insights from big data with Linguamatics I2E
Powering Knowledge Discovery Insights from big data with Linguamatics I2E Gain actionable insights from unstructured data The world now generates an overwhelming amount of data, most of it written in natural
ForeScout Open Integration Module: Data Exchange Plugin
ForeScout Open Integration Module: Data Exchange Plugin Version 3.2.0 Table of Contents About the Data Exchange Plugin... 4 Requirements... 4 CounterACT Software Requirements... 4 Connectivity Requirements...
About Intellipaat. About the Course. Why Take This Course?
About Intellipaat Intellipaat is a fast growing professional training provider that is offering training in over 150 most sought-after tools and technologies. We have a learner base of 600,000 in over
Oracle Big Data Discovery
Oracle Big Data Discovery Turning Data into Business Value Harald Erb Oracle Business Analytics & Big Data 1 Safe Harbor Statement The following is intended to outline our general product direction. It
How to choose the right approach to analytics and reporting
SOLUTION OVERVIEW How to choose the right approach to analytics and reporting A comprehensive comparison of the open source and commercial versions of the OpenText Analytics Suite In today s digital world,
DATA SCIENCE USING SPARK: AN INTRODUCTION
DATA SCIENCE USING SPARK: AN INTRODUCTION TOPICS COVERED Introduction to Spark Getting Started with Spark Programming in Spark Data Science with Spark What next? 2 DATA SCIENCE PROCESS Exploratory Data
PharmaSUG Paper CC22
PharmaSUG 2011 - Paper CC22 Importing and Parsing Comments From a PDF Document With Help From Perl Regular Expressions Joel Campbell, PPD, Inc., Wilmington, NC Ryan Wilkins, PPD, Inc., Wilmington, NC ABSTRACT
Using ElasticSearch to Enable Stronger Query Support in Cassandra
Using ElasticSearch to Enable Stronger Query Support in Cassandra www.impetus.com Introduction Relational Databases have been in use for decades, but with the advent of big data, there is a need to use
Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara
Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case
run your own search engine. today: Cablecar
run your own search engine. today: Cablecar Robert Kowalski @robinson_k http://github.com/robertkowalski Search nobody uses that, right? Services on the Market Google Bing Yahoo ask Wolfram Alpha Baidu
ExtraHop 7.3 ExtraHop Trace REST API Guide
ExtraHop 7.3 ExtraHop Trace REST API Guide 2018 ExtraHop Networks, Inc. All rights reserved. This manual in whole or in part, may not be reproduced, translated, or reduced to any machinereadable form without
Data Science. Data Analyst. Data Scientist. Data Architect
Data Science Data Analyst Data Analysis in Excel Programming in R Introduction to Python/SQL/Tableau Data Visualization in R / Tableau Exploratory Data Analysis Data Scientist Inferential Statistics &
New Oracle NoSQL Database APIs that Speed Insertion and Retrieval
New Oracle NoSQL Database APIs that Speed Insertion and Retrieval O R A C L E W H I T E P A P E R F E B R U A R Y 2 0 1 6 1 NEW ORACLE NoSQL DATABASE APIs that SPEED INSERTION AND RETRIEVAL Introduction
An Introduction to Big Data Formats
Introduction to Big Data Formats 1 An Introduction to Big Data Formats Understanding Avro, Parquet, and ORC WHITE PAPER Introduction to Big Data Formats 2 TABLE OF TABLE OF CONTENTS CONTENTS INTRODUCTION
Massive Scalability With InterSystems IRIS Data Platform
Massive Scalability With InterSystems IRIS Data Platform Introduction Faced with the enormous and ever-growing amounts of data being generated in the world today, software architects need to pay special
Stages of Data Processing
Data processing can be understood as the conversion of raw data into a meaningful and desired form. Basically, producing information that can be understood by the end user. So then, the question arises,
Using the Force of Python and SAS Viya on Star Wars Fan Posts
SESUG Paper BB-170-2017 Using the Force of Python and SAS Viya on Star Wars Fan Posts Grace Heyne, Zencos Consulting, LLC ABSTRACT The wealth of information available on the Internet includes useful and
ABSTRACT MORE THAN SYNTAX ORGANIZE YOUR WORK THE SAS ENTERPRISE GUIDE PROJECT. Paper 50-30
Paper 50-30 The New World of SAS : Programming with SAS Enterprise Guide Chris Hemedinger, SAS Institute Inc., Cary, NC Stephen McDaniel, SAS Institute Inc., Cary, NC ABSTRACT SAS Enterprise Guide (with
Exploring the Nuxeo REST API
Exploring the Nuxeo REST API Enabling Rapid Content Application Craftsmanship Copyright 2018 Nuxeo. All rights reserved. Copyright 2017 Nuxeo. All rights reserved. Chapter 1 The Nuxeo REST API What do
Oracle Big Data SQL. Release 3.2. Rich SQL Processing on All Data
Oracle Big Data SQL Release 3.2 The unprecedented explosion in data that can be made useful to enterprises from the Internet of Things, to the social streams of global customer bases has created a tremendous
BigInsights and Cognos Stefan Hubertus, Principal Solution Specialist Cognos Wilfried Hoge, IT Architect Big Data IBM Corporation
BigInsights and Cognos Stefan Hubertus, Principal Solution Specialist Cognos Wilfried Hoge, IT Architect Big Data 2013 IBM Corporation A Big Data architecture evolves from a traditional BI architecture
Big Data Hadoop Course Content
Big Data Hadoop Course Content Topics covered in the training Introduction to Linux and Big Data Virtual Machine ( VM) Introduction/ Installation of VirtualBox and the Big Data VM Introduction to Linux
CONSOLIDATING RISK MANAGEMENT AND REGULATORY COMPLIANCE APPLICATIONS USING A UNIFIED DATA PLATFORM
CONSOLIDATING RISK MANAGEMENT AND REGULATORY COMPLIANCE APPLICATIONS USING A UNIFIED PLATFORM Executive Summary Financial institutions have implemented and continue to implement many disparate applications
Paper CC16. William E Benjamin Jr, Owl Computer Consultancy LLC, Phoenix, AZ
Paper CC16 Smoke and Mirrors!!! Come See How the _INFILE_ Automatic Variable and SHAREBUFFERS Infile Option Can Speed Up Your Flat File Text-Processing Throughput Speed William E Benjamin Jr, Owl Computer
QLIK INTEGRATION WITH AMAZON REDSHIFT
QLIK INTEGRATION WITH AMAZON REDSHIFT Qlik Partner Engineering Created August 2016, last updated March 2017 Contents Introduction... 2 About Amazon Web Services (AWS)... 2 About Amazon Redshift... 2 Qlik
Powerful Insights with Every Click. FixStream. Agentless Infrastructure Auto-Discovery for Modern IT Operations
Powerful Insights with Every Click FixStream Agentless Infrastructure Auto-Discovery for Modern IT Operations The Challenge AIOps is a big shift from traditional ITOA platforms. ITOA was focused on data
Integrating SAS Analytics into Your Web Page
Paper SAS2145-2018 Integrating SAS Analytics into Your Web Page James Kochuba and David Hare, SAS Institute Inc. ABSTRACT SAS Viya adds enhancements to the SAS Platform that include the ability to access
Lambda Architecture for Batch and Stream Processing. October 2018
Lambda Architecture for Batch and Stream Processing October 2018 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document is provided for informational purposes only.
IBM Advantage: IBM Watson Compare and Comply Element Classification
IBM Advantage: IBM Watson Compare and Comply Element Classification Executive overview... 1 Introducing Watson Compare and Comply... 2 Definitions... 3 Element Classification insights... 4 Sample use cases...
Dealing with Data Especially Big Data
Dealing with Data Especially Big Data INFO-GB-2346.01 Fall 2017 Professor Norman White nwhite@stern.nyu.edu normwhite@twitter Teaching Assistant: Frenil Sanghavi fps241@stern.nyu.edu Administrative Assistant:
Xcelerated Business Insights (xbi): Going beyond business intelligence to drive information value
KNOWLEDGENT INSIGHTS volume 1 no. 5 October 7, 2011 Xcelerated Business Insights (xbi): Going beyond business intelligence to drive information value Today s growing commercial, operational and regulatory
Using SAS software to shrink the data in your applications
Paper 991-2016 Using SAS software to shrink the data in your applications Ahmed Al-Attar, AnA Data Warehousing Consulting LLC, McLean, VA ABSTRACT This paper discusses the techniques I used at the Census
The Future of Transpose: How SAS Is Rebuilding Its Foundation by Making What Is Old New Again
Paper 701-2017 The Future of Transpose: How SAS Is Rebuilding Its Foundation by Making What Is Old New Again Scott Mebust, SAS Institute Inc., Cary, NC ABSTRACT As computer technology advances, SAS continually
Tableau Server Platform Monitoring
Tableau Server Platform Monitoring Customers choose Tableau Server as their modern analytics platform because it gives their company a competitive advantage. Tableau quickly empowers users to transform
E l a s t i c s e a r c h F e a t u r e s. Contents
Elasticsearch Features A n Overview Contents Introduction... 2 Location Based Search... 2 Search Social Media(Twitter) data from Elasticsearch... 4 Query Boosting in Elasticsearch... 4 Machine Learning
CIS 612 Advanced Topics in Database Big Data Project Lawrence Ni, Priya Patil, James Tench
CIS 612 Advanced Topics in Database Big Data Project Lawrence Ni, Priya Patil, James Tench Abstract Implementing a Hadoop-based system for processing big data and doing analytics is a topic which has been
Built for Speed: Comparing Panoply and Amazon Redshift Rendering Performance Utilizing Tableau Visualizations
Built for Speed: Comparing Panoply and Amazon Redshift Rendering Performance Utilizing Tableau Visualizations Table of contents Faster Visualizations from Data Warehouses 3 The Plan 4 The Criteria 4 Learning
Is Your Data Viable? Preparing Your Data for SAS Visual Analytics 8.2
Paper SAS1826-2018 Is Your Data Viable? Preparing Your Data for SAS Visual Analytics 8.2 Gregor Herrmann, SAS Institute Inc. ABSTRACT We all know that data preparation is crucial before you can derive
A Macro to Create Program Inventory for Analysis Data Reviewer s Guide Xianhua (Allen) Zeng, PAREXEL International, Shanghai, China
PharmaSUG 2018 - Paper QT-08 A Macro to Create Program Inventory for Analysis Data Reviewer s Guide Xianhua (Allen) Zeng, PAREXEL International, Shanghai, China ABSTRACT As per Analysis Data Reviewer s
LazyBase: Trading freshness and performance in a scalable database
LazyBase: Trading freshness and performance in a scalable database (EuroSys 2012) Jim Cipar, Greg Ganger, *Kimberly Keeton, *Craig A. N. Soules, *Brad Morrey, *Alistair Veitch PARALLEL DATA LABORATORY
SAS CURRICULUM. BASE SAS Introduction
SAS CURRICULUM BASE SAS Introduction Data Warehousing Concepts What is a Data Warehouse? What is a Data Mart? What is the difference between Relational Databases and the Data in Data Warehouse (OLTP versus
Towards a Real- time Processing Pipeline: Running Apache Flink on AWS
Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Dr. Steffen Hausmann, Solutions Architect Michael Hanisch, Manager Solutions Architecture November 18 th, 2016 Stream Processing Challenges
Outrun Your Competition With SAS In-Memory Analytics Sascha Schubert Global Technology Practice, SAS
Outrun Your Competition With SAS In-Memory Analytics Sascha Schubert Global Technology Practice, SAS Topics AGENDA Challenges with Big Data Analytics How SAS can help you to minimize time to value with
APPENDIX 4 Migrating from QMF to SAS/ ASSIST Software. Each of these steps can be executed independently.
255 APPENDIX 4 Migrating from QMF to SAS/ ASSIST Software Introduction 255 Generating a QMF Export Procedure 255 Exporting Queries from QMF 257 Importing QMF Queries into Query and Reporting 257 Alternate