MySQL 5.7 and MongoDB

Transcription

1 MySQL 5.7 and MongoDB Geospatial Introduction Alexander Rubin - Principal Architect, Percona Mike Benshoof - Technical Account Manager, Percona

2 Agenda Introduction GIS in MySQL GIS in MongoDB 2

3 What is Geodata? Encompasses a wide array of topics Revolves around geo-positioning data (latitude/longitude) Point data (single lat/lon) Bounded areas (think radius from point) Defined area (think city limits outline on map) Often includes some function of distance Distance between points All points within x, y 3

4 Why do we care? Here are some commonly asked questions based around geodata: What are the 5 closest restaurants to my hotel? How far is it from here to the airport? How many restaurants are there within 25 miles of my hotel? These are all fairly common questions especially with the prevalence of geo-enabled devices (anyone here ever enable Location Services on their phone?) 4

5 Other Industries Oil/gas exploration Meteorology Logistics companies Travel companies < INSERT YOUR INDUSTRY HERE > The point: geodata is so readily available, you are likely already using it or will be soon! 5

6 High Level theory and formulas everyone s favorite Distance between points on sphere (The Haversine Formula): Everyone get out your calculators and slide rules... 6

7 En MySQL por favor... MySQL prior to 5.6 Get out your calculators MySQL 5.6 Introduced st_distance (built-in) MySQL 5.7 New spatial functions, InnoDB spatial indexes MySQL 8.0 (upcoming) The world is not flat any more (more below) 7

8 Enough with the theory already! = 37; = -133; = 38; = -133; SELECT (3959 * acos(cos(radians(@src_lat)) * cos(radians(@dest_lat)) * cos(radians(@dest_lng) - radians(@src_lng)) + sin(radians(@src_lat)) * sin( radians(@dest_lat)))) as GreatCircleDistance GreatCircleDistance Huzzah!! The distance between two latitude lines is ~69 miles! 8

9 now let s put it all together InnoDB Table Structure (pre 5.6) Find me all zip codes within 25 miles of my current zip code... CREATE TABLE `zipcodes` ( `id` int(11) NOT NULL AUTO_INCREMENT, `zip` varchar(255) DEFAULT NULL, `latitude` decimal(10,8) DEFAULT NULL, `longitude` decimal(11,8) DEFAULT NULL, `city` varchar(255) DEFAULT NULL, `state` varchar(255) DEFAULT NULL, `county` varchar(255) DEFAULT NULL, `type` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), KEY `lat` (`latitude`,`longitude`), KEY `city` (`city`,`state`), KEY `state` (`state`), KEY `zip` (`zip`), KEY `county` (`county`) ) ENGINE=InnoDB 9

10 add some help Helper Function (pre 5.6) DELIMITER $$ DROP FUNCTION IF EXISTS DistanceInMiles$$ CREATE FUNCTION DistanceInMiles (src_lat decimal(10,8), src_lng decimal(11,8), dest_lat decimal(10,8), dest_lng decimal(11,8)) RETURNS decimal(15,8) DETERMINISTIC BEGIN RETURN CAST((3959 * acos(cos(radians( src_lat)) * cos(radians( dest_lat)) * cos(radians( dest_lng) - radians( src_lng)) + sin( radians( src_lat)) * sin( radians( dest_lat)))) as decimal(15,8)); END $$ DELIMITER ; 10

11 and results! mysql> # Norman, OK Post Office (73071) mysql> = ; Query OK, 0 rows affected (0.00 sec) mysql> = ; Query OK, 0 rows affected (0.00 sec) mysql> = 25; Query OK, 0 rows affected (0.00 sec) mysql> mysql> SELECT z.zip, z.city, z.state, -> z.latitude, z.longitude) as distance -> FROM zipcodes z -> HAVING distance -> ORDER BY distance -> LIMIT 10; ZIP City State distance NORMAN OK NORMAN OK NORMAN OK OKLAHOMA CITY OK NORMAN OK NOBLE OK OKLAHOMA CITY OK NORMAN OK NORMAN OK OKLAHOMA CITY OK rows in set (0.47 sec) 11

12 Not so great... mysql> EXPLAIN SELECT z.zip, z.city, z.state, z.latitude, z.longitude) as distance -> FROM zipcodes z -> HAVING distance -> ORDER BY distance -> LIMIT 10\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: z partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: filtered: Extra: Using temporary; Using filesort 1 row in set, 1 warning (0.00 sec) 12

13 More math to the rescue! Rather than scan the whole table, let s just look at a small rectangle of data (i.e. a bounding box): 1 o Latitude ~= 69 miles 1 o Longitude ~= cos(lat) * 69 $lat_range = radius / 69 $lng_range = abs(radius / (cos(lat) * 69)) $lon1 = $mylng + $lng_range $lon2 = $mylng - $lng_range $lat1 = $mylat +$lat_range $lat2 = $mylat - $lat_range 13

14 and respectable! mysql> SELECT z.zip, z.city, z.state, z.latitude, z.longitude) as distance -> FROM zipcodes z -> WHERE z.longitude BETWEEN AND > AND z.latitude BETWEEN AND > HAVING distance -> ORDER BY distance -> LIMIT 10; ZIP City State distance NORMAN OK NORMAN OK NORMAN OK OKLAHOMA CITY OK NORMAN OK NOBLE OK OKLAHOMA CITY OK NORMAN OK NORMAN OK OKLAHOMA CITY OK rows in set (0.00 sec) Updated Explain Plan: id: 1 select_type: SIMPLE table: z partitions: NULL type: range possible_keys: lat key: lat key_len: 13 ref: NULL rows: 1616 filtered: Extra: Using index condition; Using temporary;... 14

15 15 Onwards and upwards to Spatial Data Types!

16 Converting our table to geospatial... Non-Spatial way: Lat / Lon as individual decimal columns Composite B-Tree index covering each column Spatial way: Use the geometry type with a POINT object (single lat/lon) Use the geometry type with a POLYGON object (defined border) Use a SPATIAL index on the geometry column (MyISAM only through 5.6, added to InnoDB in 5.7) 16

17 And voila! CREATE TABLE `legacy`.`zipcodes` ( `id` int(11) NOT NULL AUTO_INCREMENT, `zip` varchar(255) DEFAULT NULL, `latitude` decimal(10,8) DEFAULT NULL, `longitude` decimal(11,8) DEFAULT NULL, `city` varchar(255) DEFAULT NULL, `state` varchar(255) DEFAULT NULL, `county` varchar(255) DEFAULT NULL, `type` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), KEY `lat` (`latitude`,`longitude`), KEY `city` (`city`,`state`), KEY `state` (`state`), KEY `zip` (`zip`), KEY `county` (`county`) ) ENGINE=InnoDB CREATE TABLE `geospatial`.`zipcodes` ( `id` int(11) NOT NULL AUTO_INCREMENT, `zip` varchar(255) DEFAULT NULL, `geo` geometry NOT NULL, `city` varchar(255) DEFAULT NULL, `state` varchar(255) DEFAULT NULL, `county` varchar(255) DEFAULT NULL, `type` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), SPATIAL KEY `geo` (`geo`), KEY `city` (`city`,`state`), KEY `state` (`state`) ) ENGINE=InnoDB Now, we can use GIS notation to find our zip code: SELECT zip, city, state, county, type, ST_AsText(geo) lon_lat FROM zipcodes WHERE ST_CONTAINS(geo, ST_GeomFromText('POINT( )', 4326)); 17

18 So What?! We had already written a query do just that And this one is still doing the same amount of handler operations! This opens up the opportunity to run other GIS based calculations as we'll see now... 18

19 Not just points! In earlier slides, we based everything on a single lat/lon coordinate Several zip code databases follow this model In recent years, the prevalence of full polygon region databases has increased Example: the US Census provided tl_2014_us_zcta510 schema... 19

20 Not just points! This table defines full regions for postal codes as opposed to a centered lat/lon for an actual building: 20 CREATE TABLE `zipcodes` ( `ogr_fid` int(11) NOT NULL AUTO_INCREMENT, `shape` geometry NOT NULL, `zcta5ce10` varchar(5) DEFAULT NULL, `geoid10` varchar(5) DEFAULT NULL, `classfp10` varchar(2) DEFAULT NULL, `mtfcc10` varchar(5) DEFAULT NULL, `funcstat10` varchar(1) DEFAULT NULL, `aland10` double DEFAULT NULL, `awater10` double DEFAULT NULL, `intptlat10` varchar(11) DEFAULT NULL, `intptlon10` varchar(12) DEFAULT NULL, PRIMARY KEY (`ogr_fid`), SPATIAL KEY `shape` (`shape`), KEY `zipcode` (`zcta5ce10`) ) ENGINE=InnoDB

21 Difference in storage Old version for Norman zip code 73071: ST_AsText(geo): POINT( ) New version in polygon based version: ST_AsText(shape): POLYGON(( , , , , , , , , , , , , , , , , , , ,... Note that now, we have a full region as opposed to a point... 21

22 GIS Functions st_contains Check if an object is entirely within another object st_within Check if an object is spatially within another object st_* Detailed list here: 22

23 Some Sample GIS Queries # Find me all the zipcodes (polygons) for my points of interest SELECT zip.zcta5ce10 AS zipcode, pts.name FROM my_points pts JOIN zipcodes zip ON st_within(pts.loc, zip.shape); # Find me all points of interest within a zip code (polygon) SELECT zip.zcta5ce10 AS zipcode, pts.name FROM my_points pts JOIN zipcodes zip ON st_contains(zip.shape, pts.loc) WHERE zip.zcta5ce10 = "73071"; 23

24 Some Sample GIS Queries # Find me all the points of interest within a county SELECT zip.zcta5ce10 AS zipcode, pts.name FROM my_points pts JOIN zipcodes zip ON st_contains(zip.shape, pts.loc) JOIN legacy.zipcodes leg ON leg.zip = zip.zcta5ce10 WHERE leg.county = "Cleveland"; 24

25 Impact of SPATIAL KEY I mentioned earlier that SPATIAL keys are available for InnoDB starting in 5.7 Here is just quick sample of the same query, with and without a SPATIAL key around a polygon Let s grab one of the earlier queries: SELECT raw.zcta5ce10 AS zipcode, pts.name FROM my_points pts JOIN tl_2014_us_zcta510 raw ON st_within(pts.loc, raw.shape); 25

26 Impact of SPATIAL KEY Without Key: zipcode name POI POI rows in set (57.38 sec) Variable_name Value Handler_read_first 2 Handler_read_key 2 Handler_read_last 0 Handler_read_next 0 Handler_read_prev 0 Handler_read_rnd 0 Handler_read_rnd_next Handler_write With Key: zipcode name POI POI rows in set (0.00 sec) Variable_name Value Handler_read_first 1 Handler_read_key 4 Handler_read_last 0 Handler_read_next 4 Handler_read_prev 0 Handler_read_rnd 0 Handler_read_rnd_next 4 Handler_write

27 MySQL 8 - GIS Updates Spatial indexes for InnoDB (added in 5.7) ST_Distance() returns geodetic distance on the ellipsoid in meters! other functions still need helper functions (st_length() for example), but are being set to be updated in future versions ST_AsText() Returns lat/lon order based on spatial reference system MySQL now stores information about spatial reference systems other than SRID 0, for use with spatial data Note - all previous slides used SRID 4326 (geographical coordinates)

28 Quick Demo Collect points along a trip (GPX format) Load the raw file Determine geodata about each point Calculate total distance driven per county

29 Quick Demo mysql> explain SELECT raw.zcta5ce10 AS zipcode, raw.city, raw.county, raw.state, st_astext(pts.pt) FROM trip_tracker.raw_trip_points pts JOIN geopoly.geo_zips raw ON st_contains(raw.shape, pts.pt) WHERE pts.trip_id = 16\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: pts partitions: NULL type: ref possible_keys: loc,trip_idx key: trip_idx key_len: 4 ref: const rows: 5321 filtered: Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: raw partitions: NULL type: ALL possible_keys: SHAPE key: NULL key_len: NULL ref: NULL rows: filtered: Extra: Range checked for each record (index map: 0x2)

30 Quick Demo Variable_name Value Handler_commit 1 Handler_delete 0 Handler_discover 0 Handler_external_lock 4 Handler_mrr_init 0 Handler_prepare 0 Handler_read_first 0 Handler_read_key 5323 Handler_read_last 0 Handler_read_next Handler_read_prev 0 Handler_read_rnd 0 Handler_read_rnd_next 0 Handler_rollback 0 Handler_savepoint 0 Handler_savepoint_rollback 0 Handler_update 0 Handler_write

31 MySQL 5.7: GeoJSON and Points of Interests ; = ; = ' ')'), 1); SELECT CONCAT('{ "type": "FeatureCollection", "features": [ ', GROUP_CONCAT('{ "type": "Feature", "geometry": ', ST_AsGeoJSON(shape), ', "properties": {"distance":', ', "name":"', name, '"} }' order by '] }') as j FROM points_new WHERE st_within(shape, 10)) and (other_tags like '%"amenity"=>"cafe"%' or other_tags like '%"amenity"=>"restaurant"%') and name is not null and < 450 limit 10;

32 Walk to lunch from Percona office

33 Points of Interest: data Openstreatmap: $ ogr2ogr -overwrite -progress -f "MySQL" MYSQL:osm,user=root north-america-latest.osm.pbf

34 Points of Interest: data mysql> select name, st_astext(shape), other_tags from points_new where other_tags like '%"amenity"=>"cafe"%' or other_tags like '%"amenity"=>"restaurant"%' and name is not null limit 10\G *************************** 1. row *************************** name: The Widow's Tavern and Grille st_astext(shape): POINT( ) other_tags: "amenity"=>"restaurant","old_name"=>"widow Brown's Tavern"

35 MySQL 5.7: Creating bounding box rectangle CREATE DEFINER = current_user() FUNCTION create_envelope(lat decimal(20, 14), lon decimal(20, 14), dist int) RETURNS geometry DETERMINISTIC begin declare point_text varchar(255); declare l varchar(255); declare p geometry; declare env geometry; declare rlon1 double; declare rlon2 double; declare rlat1 double; declare rlat2 double; set point_text = concat('point(', lon, ' ', lat, ')'); set p = ST_GeomFromText(point_text, 1); set rlon1 = lon-dist/abs(cos(radians(lat))*69); set rlon2 = lon+dist/abs(cos(radians(lat))*69); set rlat1 = lat-(dist/69); set rlat2 = lat+(dist/69); set l = concat('linestring(', rlon1, ' ', rlat1, ',', rlon2, ' ', rlat2, ')'); set env= ST_Envelope(ST_GeomFromText(l, 1)); return env; end //

36 MySQL Spatial Indexes in 5.7 No create_envelope mysql> explain SELECT.. FROM points_new WHERE (other_tags like '%"amenity"=>"cafe"%' or other_tags like '%"amenity"=>"restaurant"%') and name is not null and < 450 limit 10\G *************************** 1. row*************************** id: 1 select_type: SIMPLE table: points_new partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: filtered: Extra: Using where With create_envelope mysql> explain SELECT.. FROM points_new WHERE st_within(shape, 10)) (other_tags like '%"amenity"=>"cafe"%' or other_tags like '%"amenity"=>"restaurant"%') and name is not null and < 450 limit 10\G *************************** 1. row*************************** id: 1 select_type: SIMPLE table: points_new partitions: NULL type: range possible_keys: SHAPE key: SHAPE key_len: 34 ref: NULL rows: 665 filtered: Extra: Using where

37 Entering geonear Performs a geospatial query that returns the documents closest to a given point. geosearch Performs a geospatial query that uses MongoDB s haystack index functionality. Is not needed in Mongo...

38 From to MySQL 5.7 MongoDB 3.2 SELECT osm_id, name, round(st_distance_sphere(shape, st_geomfromtext('point ( )', 1) ), 2) as dist, st_astext(shape) FROM points_new WHERE 10)) and (other_tags like '%"amenity"=>"cafe"%' or other_tags like '%"amenity"=>"restaurant"%') and name is not null ORDER BY dist asc LIMIT 10; db.runcommand( { geonear: "points", near: { type: "Point", coordinates: [ , ] }, spherical: true, query: { name: { $exists: true, $ne:null}, "other_tags": { $in: [ /.*amenity=>restaurant.*/, /.*amenity=>cafe.*/ ] } }, "limit": 5, "maxdistance": } )

39 From to MySQL 5.7 MongoDB 3.2 mysql> SELECT osm_id, name, round(st_distance_sphere(shape, st_geomfromtext('point... -> ORDER BY dist asc LIMIT 10; osm_id name dist st_astext(shape) Pop's POINT( ) toast POINT( ) Pizzeria Toro POINT( ) Parker & Otis POINT( ) Torero's POINT( ) Fishmonger's POINT( ) Lilly's Pizza POINT( ) Dame's Chicken and Waffles POINT( ) El Rodeo POINT( ) Piazza Italia POINT( ) rows in set (0.21 sec) db.runcommand( { geonear: "points",... } ) Milliseconds: "stats" : { "nscanned" : 1728, "objectsloaded" : 1139, "avgdistance" : , "maxdistance" : , "time" : 17 }, "ok" : 1

40 From to Export from MySQL 5.7: mysql> SELECT JSON_OBJECT('name', replace(name, '"', ''), 'other_tags', replace(other_tags, '"', ''), 'geometry', st_asgeojson(shape)) as j FROM `points` INTO OUTFILE '/var/lib/mysql-files/points.json'; Query OK, rows affected (4 min 1.35 sec)

41 From to Load to MongoDB (parallel): mongoimport --db osm --collection points -j 24 --file /var/lib/mysql-files/points.json T22:38: connected to: localhost T22:38: [...] osm.points 31.8 MB/2.2 GB (1.4%) T22:38: [...] osm.points 31.8 MB/2.2 GB (1.4%) T22:38: [...] osm.points 31.8 MB/2.2 GB (1.4%) T23:12: [########################] osm.points 2.2 GB/2.2 GB (100.0%) T23:12: imported documents

42 From to Points json example (OpenStreetMap data) { "name": "Wendy's", "geometry": { "type": "Point", "coordinates": [ , ] }, "other_tags": "amenity=>fast_food" } GeoJSON

43 From to Creating index > use osm switched to db osm > db.points.createindex({ geometry : "2dsphere" } ) { "createdcollectionautomatically" : false, "numindexesbefore" : 1, "numindexesafter" : 2, "ok" : 1 }

44 From to Now trying lines collection ooops! > db.lines.createindex({ geometry : "2dsphere" } ) { "createdcollectionautomatically" : false, "numindexesbefore" : 1, "errmsg" : "exception: Can't extract geo keys: { _id: Same coordinates for start and end ObjectId(' f45f7f0d6dfbed2'), name: \"75 North\", geometry: { type: \"LineString\", coordinates: [ [ , ], [ , ] ] }, other_tags: \"tiger:cfcc=>a41,tiger:county=>kosciusko, IN,tiger:name_base=>75,tiger:name_direction_suffix=>N,tiger:reviewed=>no\" } GeoJSON LineString must have at least 2 vertices: [ [ , ], [ , ] ]", "code" : 16755, "ok" : 0 }

45 From to Removing the bad data is painful and slow but fixed the issue > db.lines.remove({"geometry.type": "LineString", "geometry.coordinates": {$size:2}, $where: "this.geometry.coordinates[0][0] == this.geometry.coordinates[1][0] && this.geometry.coordinates[0][1] == this.geometry.coordinates[1][1]" }) WriteResult({ "nremoved" : 22 }) $where is slow and acquire global lock

46 Find all cafes near our Durham, NC office: > db.runcommand( { geonear: "points", near: { type: "Point", coordinates: [ , ]}, spherical: true,... query: { "other_tags": { $in: [ /.*amenity=>restaurant.*/, /.*amenity=>cafe.*/ ] } }, "limit": 5, "maxdistance": } ) { "results" : [ { "dis" : , "obj" : { "_id" : ObjectId(" f45f7f0d66f8f13"), "name" : "Pop's", "geometry" : { "type" : "Point", "coordinates" : [ , ] }, "other_tags" : "addr:city=>durham,addr:country=>us,addr:ho usenumber=>605, addr:street=>west Main Street,amenity=>restaurant,building=>yes" }...

47 MySQL 5.7 and MongoDB: Geospatial Introduction Questions? 47

48 MySQL 5.7 and MongoDB: Geospatial Introduction Thanks! 48