Spectrum Version Release Notes

Transcription

1 Spectrum Spatial for Big Data Version Release Notes This document contains the new and updated features for Spectrum Spatial for Big Data. Contents: What's New? 2 Fixed Issues 3 Known Issues 3 System Requirements and Dependencies 5

2 What's New? The Spectrum Spatial for Big Data version suite of products (Spectrum Geocoding for Big Data, Spectrum Location Intelligence for Big Data, and Spectrum Routing for Big Data) contains the following new and updated features: Geocoding: Improvements in the geocoder to provide the highest match rate. Improved parsing logic for Spark and MapReduce job parameters. Enhancements in the Spark and MapReduce job configuration to support command line jobs parameters. Enhancements in the Spark and MapReduce job configuration handler. Improved CSV reader. The CSV reader supports a configurable delimiter, quote, and escape character. For backward compatibility, configure the escape character as a backslash. Spectrum Geocoding for Big Data now supports the Canada MLD dataset along with the USA MLD dataset. See the Spectrum Geocoding for Big Data User Guide for more information. For the geocoding drivers for Spark and MapReduce (both geocode and reverse geocode), we have standardized on the Hadoop "-conf" option rather than "-config" which supports multiple conf files as well as the ability to easily specify additional variables with the -Doption=value syntax. For Spark and MapReduce jobs, multiple input column indexes are supported for any given geocode field (except the countrycolumn field which allows only a single entry). The new lastlinecolumn field supports specifying the last line information of an address as a single entity rather than requiring a combination of City/State/Postcode. The streetnamecolumn field can be used to specify unparsed address information rather than forcing it to be broken into address number, street name, unit number. For Hive user-defined functions, the following signatures are now supported: geocode(singlelineaddress,country) geocode(addressfirstline,addresslastline,country) geocode(addressnumber,street,unitnumber,city,state,postcode,country) Routing: GRA Java library (v 2.0.4) is included in the Routing distribution zip. LI SDK: LI SDK Java library (v 1.3.0) is included in the Location Intelligence distribution zip. Library Upgrades: Global Geocoding API (GGA) Location Intelligence SDK (LI SDK) See the Spectrum Spatial for Big Data User Guides for more information on these features. Spectrum Spatial for Big Data Release Notes 2

3 Fixed Issues (HAD-1737) For the Spark HexAggregation sample, all the fields in the output CSV will be separated by the same delimiter which is provided via command line. Before this fix, only the first field was separated by the provided delimiter and other fields were always separated by a tab character. Before: (delimiter ' '): PF381 POLYGON (( ,...)) After: (delimiter ' '): PF381 POLYGON (( ,...)) Known Issues (HAD-2294) Parameter values in the Geocoding configuration file should have trimmed values. There must be no leading or preceding space characters, especially in the specified paths. (HAD-1397) Each session of HiverServer2 clients might create an instance of the Routing engine which may lead to multiple instances to be in an open state. Workaround: Add routing hive jar to HIVE_AUX_JARS_PATH. Refer to the Spectrum Routing for Big Data User Guide. (HAD-1369) UDFs that take WritableGeometry as argument, for example, SquareHashBoundary GeoHashBoundary, HexagonBoundary, or ToWKT may throw the following exception with Hive on Cloudera environment (CDH 5.9 and 5.10.) when an invalid input is given in argument instead of a valid WritableGeometry. FAILED: ClassCastException org.apache.hadoop.hive.serde2.typeinfo.structtypeinfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.primitivetypeinfo Examples SquareHashBoundary given an invalid coordinate as below: SELECT ToWKT(SquareHashBoundary("-73","242",3)); ToWKT with invalid WritableGeometry returned from FromWKT UDF. SELECT ToWKT(FromWKT("", "epsg:4326")); Workaround: Set hive.optimize.constant.propagation=false to remove this error. Spectrum Spatial for Big Data Release Notes 3

4 (HAD-1251) Geocoding UDF return "data not available" error in second or subsequent sessions with Hive Server2 remote client/s, for example, Beeline or any JDBC client. Currently the error is reproducible with TEZ as the Hive engine. Workaround: Add geocoding Hive jar to HIVE_AUX_JARS_PATH. Refer to Spectrum Geocoding for Big Data User Guide. Spectrum Spatial for Big Data Release Notes 4

5 System Requirements and Dependencies Spectrum Spatial for Big Data is collection of jar files that can be deployed to your Hadoop system. Note: Installation is not required. These jar files are compatible with any Hadoop Distribution having: Hadoop version 2.6 and above Spark version and above Note: This product is verified on the following Hadoop Distributions. Hortonworks Data Platform 2.1 on Red Hat Linux 6.3 Hortonworks Data Platform 2.4 on CentOS 7.0 Hortonworks Data Platform 2.6 on CentOS 7.0 Cloudera CDH 5.10 on CentOS 7.3 Cloudera CDH 5.11 on CentOS 7.3 Amazon EMR 5.5 Amazon EMR 5.6 To use these jar files, you must be familiar with configuring Hadoop in Hortonworks, Cloudera,or EMR and developing applications for distributed processing. For more information, refer to Hortonworks, Cloudera, or EMR documentation. To use the product, the following must be installed on your system: for MapReduce: Java JDK version 1.7 or above Hadoop version or above for Spark: Java JDK version 1.7 or above Hadoop version or above Spark version or above for Hive: Hive version or above for Hive Client For example, Beeline Spectrum Spatial for Big Data Release Notes 5

6 for Zeppelin Notebook (not supported in Cloudera) Java JDK version 1.7 or above Hadoop version or above Spark version 2.0 or above Spectrum Spatial for Big Data Release Notes 6

7 3001 Summer Street Stamford CT USA Pitney Bowes Software Inc. All rights reserved