Feature Cache Framework for GeoTools

Size: px

Start display at page:

Download "Feature Cache Framework for GeoTools"

Felix Hensley
5 years ago
Views:

1 Feature Cache Framework for GeoTools A Google Summer of Code 2007 Project Christophe Rousson, Université Laval project mentored by Ian Turton 1

2 Outline Project motivations Benchmarks Future work GeoTools Data Access Model Cache Framework Design

3 Outline Project motivations Benchmarks Future work GeoTools Data Access Model Cache Framework Design

4 Project Motivations

5 Project Motivations Client side

6 Project Motivations Client side Don t download same data again and again

7 Project Motivations Client side Save bandwidth Don t download same data again and again

8 Project Motivations Client side Save bandwidth Don t download same data again and again Ability to disconnect from network

9 Outline Project motivations Benchmarks Future work GeoTools Data Access Model Cache Framework Design

10 GeoTools Data Module

11 Feature a common interface to encapsulate geographic vector data based on the OGC Reference Model SimpleFeature sub-interface (OGC SFS)

12 DataStore vs. FeatureSource

13 DataStore vs. FeatureSource Low-level API DataStore

14 DataStore vs. FeatureSource Low-level API High-level API DataStore

15 DataStore vs. FeatureSource Low-level API DataStore High-level API FeatureSource (read-only)

16 DataStore vs. FeatureSource Low-level API DataStore High-level API FeatureSource (read-only) Transaction FeatureStore (read-write, transactional)

17 DataStore vs. FeatureSource Low-level API DataStore High-level API FeatureSource (read-only) Transaction FeatureStore (read-write, transactional) LockingManager FeatureLocking

18 WFS DataStore Example HashMap params = new HashMap(); WFSDataStoreFactory factory = new WFSDataStoreFactory(); try { params.put(wfsdatastorefactory.url.key, new URL(" version=1.0.0&request=getcapabilities&service=wfs")); DataStore ds = (WFSDataStore) factory.createnewdatastore(params); FeatureSource source = ds.getfeaturesource("road"); FeatureCollection fc = source.getfeatures(filter); } catch (MalformedURLException e) { //... } catch (IOException e) { //... }

19 Supported DataStores Shapefile Postgis, Oracle, DB2 WFS, ArcSDE many other unsupported data plugins

20 Outline Project motivations Benchmarks Future work GeoTools Data Access Model Cache Framework Design

21 Cache basics CLIENT SOURCE

22 Cache basics CLIENT SOURCE get data

23 Cache basics CLIENT SOURCE

24 Cache basics CLIENT SOURCE get

25 Cache basics CLIENT CACHE STORAGE SOURCE get

26 Cache basics CLIENT CACHE STORAGE SOURCE get peek

27 Cache basics CLIENT CACHE STORAGE SOURCE get peek data

28 Cache basics CLIENT CACHE STORAGE SOURCE get peek data get data

29 Cache basics CLIENT CACHE STORAGE SOURCE get peek data get data store

30 Cache basics CLIENT CACHE STORAGE SOURCE get peek data get data store data

31 Requirements optimized for how features are usually accessed

32 Filter

33 Rewrite Filter = BBox AND Filter

34 Rewrite Filter = BBox AND Filter Cache nodes

35 Rewrite Filter = BBox AND Filter Cache nodes Blocks of Data

36 Storage

37 ? Storage

38 ? associate with Storage

39 Divide space into regions Storage? associate with

40 Divide space into regions Storage? associate with Group and store features by regions

41 Divide space into regions Storage? Node ID Group and store features by regions

42 Divide space into regions Storage? Node ID Group and store features by regions

43 Divide space into regions Storage? Node ID Known & unknown regions Valid & invalid nodes Group and store features by regions

44 Divide space into regions Storage? Node ID Known & unknown regions Valid & invalid nodes Group and store features by regions

45 How to divide space into regions?

46 How to divide space into regions? regular space division grid

47 How to divide space into regions? regular space division grid regular space division quadtree

48 How to divide space into regions? regular space division grid regular space division quadtree irregular space division R-tree

49 How to divide space into regions? most efficient regular space division grid regular space division quadtree irregular space division R-tree

50 How to divide space into regions? most simple most efficient regular space division grid regular space division quadtree irregular space division R-tree

51 Feature Cache Framework FeatureCache

52 Feature Cache Framework FeatureCache Storage

53 Feature Cache Framework FeatureCache Tracker Storage

54 Feature Cache Framework FeatureCache EvictionPolicy Tracker Storage

55 Feature Cache Framework FeatureCache EvictionPolicy Tracker Storage

56 Feature Cache Framework FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache EvictionPolicy LRUEvictionPolicy

57 Feature Cache Framework FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache EvictionPolicy LRUEvictionPolicy

58 Feature Cache Framework FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache EvictionPolicy LRUEvictionPolicy

59 Feature Cache Framework FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache EvictionPolicy LRUEvictionPolicy

60 Feature Cache Framework FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache EvictionPolicy LRUEvictionPolicy

61 Using spatial indexation spatial index framework proposed by Marios Hadjieleftheriou spatialindex

62 Example code int feature_type_index = 0; int number_of_nodes = 500; int cache_feature_capacity = 1000; DataStore ds =... ; // get any instance of a datastore FeatureCache cache = new GridFeatureCache( ds.getfeaturesource( ds.gettypenames([feature_type_index]), number_of_nodes, cache_feature_capacity, BufferedDiskStorage.createInstance()); // cache may be used anywhere you would normally use FeatureSource

63 Extension points FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache EvictionPolicy LRUEvictionPolicy

64 Extension points FeatureSource SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache 1 Inner mechanisms EvictionPolicy LRUEvictionPolicy

65 Extension points FeatureSource Type of index 2 SpatialIndex Storage FeatureCache FeatureListener Tracker EvictableTree AbstractFeatureCache 1 Inner mechanisms EvictionPolicy LRUEvictionPolicy

66 Extension points FeatureSource Type of index 2 SpatialIndex Storage 3 FeatureCache FeatureListener Tracker Storage/ Backend EvictableTree AbstractFeatureCache 1 Inner mechanisms EvictionPolicy LRUEvictionPolicy

67 Extension points FeatureSource Type of index 2 SpatialIndex Storage 3 FeatureCache FeatureListener Tracker Storage/ Backend EvictableTree AbstractFeatureCache 1 Inner mechanisms EvictionPolicy LRUEvictionPolicy 4 Eviction policy

68 Project Current State

69 Project Current State Grid Spatial Index

70 Project Current State Grid Spatial Index Memory and Disk Storage

71 Project Current State Grid Spatial Index Memory and Disk Storage Thread safe

72 Project Current State Grid Spatial Index Memory and Disk Storage Thread safe LRU eviction policy

73 Project Current State Grid Spatial Index Memory and Disk Storage Thread safe LRU eviction policy Eviction based on a maximum number of features

74 Project Current State Grid Spatial Index Memory and Disk Storage Thread safe LRU eviction policy Eviction based on a maximum number of features Reusable framework

75 DEMO

76 Outline Project motivations Benchmarks Future work GeoTools Data Access Model Cache Framework Design

77 Test Case 5000 random features in the unit square, mean size of queries query window size of 0.05, 0.02, 0.01 queries in the neighborhood of previous one

78 Relative execution time difference compared to MemoryDataStore (%) Mean 100% 75% 50% 25% 0% -25% -50% -75% -100% 73% 74% 49% -50% 300 f. 500 f f f.

79 Relative execution time difference compared to MemoryDataStore (%) Mean 100% 75% 50% 25% 0% -25% -50% -75% -100% 91% 84% 80% 77% 50 nodes 100 nodes 250 nodes 500 nodes

80 Outline Project motivations Benchmarks Future work GeoTools Data Access Model Cache Framework Design

81 Current limitations

82 Current limitations support only for SimpleFeature

83 Current limitations support only for SimpleFeature per feature type cache

84 Current limitations support only for SimpleFeature per feature type cache read-only implementation

85 TODO LIST

86 write-thru TODO LIST

87 write-thru disconnectable TODO LIST

88 TODO LIST write-thru disconnectable warm start from storage

89 TODO LIST write-thru disconnectable warm start from storage support for other types of index : quadtree, r-tree

90 Wish list

91 Wish list cache listens to update feeds

92 Wish list cache listens to update feeds cache downloads pre-built index from third-party service (r-tree case)

93 Questions? For more information, please contact : christophe.rousson@gmail.com

94 Relative execution time difference compared to MemoryDataStore (%) Worst case Optimal case Mean 100% -100% -329% 79% -50% -369% 79% 49% -334% 78% 73% -332% 79% 74% -300% -500% -700% 300 f. 500 f f f.

95 Relative execution time difference compared to MemoryDataStore (%) Worst case Optimal case Mean 100% 95% 84% 96% 91% 97% 80% 96% 77% -100% -684% -230% -340% -255% -300% -500% -700% 50 nodes 100 nodes 250 nodes 500 nodes

97 Visitor QueryStrategy Statistics SpatialIndex + insertdata() : void + deletedata() : void + clear() : void + intersectionquery() : void + containmentquery() : void + querystrategy() : void + getstatistics() : Statistics + getindexproperties() : void SimpleFeatureMarshaller Externalizable Data GridData 0..* contains 1..* 1 SimpleFeature Shape Entry 0..* 1 stores 1 Node GridNode GridRootNode has root 1 Serializable 1 1 AbstractSpatialIndex 1 Storage references children Serializable MemoryStorage 1 0..* Shape Region NodeIdentifier 0..* lists Storage DiskStorage Grid 1 1 RegionNodeIdentifier BufferedDiskStorage

GeoTools Data Store Performance and Recommendations

GeoTools Data Store Performance and Recommendations Submitted To: Program Manager GeoConnections Victoria, BC, Canada Submitted By: Jody Garnett Refractions Research Inc. Suite 400 1207 Douglas St. Victoria,