Key/Value Pair versus hstore - Benchmarking Entity-Attribute-Value Structures in PostgreSQL.

Size: px

Start display at page:

Download "Key/Value Pair versus hstore - Benchmarking Entity-Attribute-Value Structures in PostgreSQL."

Elmer Wood
5 years ago
Views:

1 Key/Value Pair versus hstore - Benchmarking Entity-Attribute-Value Structures in PostgreSQL. Michel Ott June 7, 2011 University of Applied Science Rapperswil 1

2 What is KVP? A key-value pair (KVP) is a set of two linked data items: a key, which is a unique identifier for some item of data, and the value, which is either the data that is identified or a pointer to the location of that data. (Source: Techtarget, an open-ended data structure that allows for future extension without modifying existing code or data. (Source: Wikipedia, 1. { data : [ 2. { amenity : restaurant, 3. name : Godman }, 4. { amenity : university, 5. name : Harvard University } 6. ] } June 7, 2011 University of Applied Science Rapperswil 2

3 Agenda Introduction What is hstore? KVP Schema in PostgreSQL Setup / Environment Performance Benchmark Design Test Environment Benchmark (May 2011) Results Findings Conclusion June 7, 2011 University of Applied Science Rapperswil 3

4 INTRODUCTION What is hstore? KVP Schema in PostgreSQL June 7, 2011 University of Applied Science Rapperswil 4

5 What is hstore? Hstore in PostgreSQL Storage for semistructural data (a'la perl hash) Stores associative arrays in a attribute of a table Is an abstract data type in PostgreSQL Provides a bunch of PostgreSQL functions for querying, transforming, manipulating, Usage of hstore 1. CREATE TABLE bench_hstore ( id BIGINT PRIMARY KEY, ( ), kvp_hstore HSTORE ); June 7, 2011 University of Applied Science Rapperswil 5

); id : BIGINT kvp_hstore : HSTORE 1 amenity => restaurant, name => Goodman 1.

6 What is hstore? Usage of hstore 1. INSERT INTO bench_hstore(kvp_hstore) VALUES ( hstore( amenity => restaurant name => Godman ) ); id : BIGINT kvp_hstore : HSTORE 1 amenity => restaurant, name => Goodman 1. SELECT hstore(kvp_hstore)-> name as name FROM bench_hstore WHERE hstore(kvp_hstore)-> amenity = restaurant ; June 7, 2011 University of Applied Science Rapperswil 6

7 KVP Schema in PostgreSQL Schema Two tables needed (one for the unforeseen arbitrary data = KVP and one for the additional data) Usage 1. CREATE TABLE bench_kvp_info( id BIGINT PRIMARY KEY, ( ) ); 2. CREATE TABLE bench_kvp( id BIGINT REFERENCES bench_kvp_main(id), key TEXT NOT NULL, value TEXT ); June 7, 2011 University of Applied Science Rapperswil 7

INSERT INTO bench_kvp(id, key, value) VALUES(1, name, Godman ); id : BIGINT key : TEXT value : TEXT 1 amenity

8 KVP Schema in PostgreSQL Usage 1. INSERT INTO bench_kvp_info(id) VALUES(1) 2. INSERT INTO bench_kvp(id, key, value) VALUES(1, amenity, restaurant ); 3. INSERT INTO bench_kvp(id, key, value) VALUES(1, name, Godman ); id : BIGINT key : TEXT value : TEXT 1 amenity restaurant 1 name Godman 1. SELECT * FROM bench_kvp WHERE id = ( SELECT bench_id FROM bench_kvp WHERE key = amenity' AND value = restaurant ); June 7, 2011 University of Applied Science Rapperswil 8

9 SETUP / ENVIRONMENT Performance Benchmark Design Table Schemas Test Data Test Environment June 7, 2011 University of Applied Science Rapperswil 9

10 Performance Benchmark Design Table Schema 1. CREATE TABLE bench_kvp_info( id BIGINT PRIMARY KEY, ( ) ); 2. CREATE TABLE bench_kvp( id BIGINT REFERENCES bench_kvp_main(id), key TEXT NOT NULL, value TEXT ); 1. CREATE TABLE bench_hstore ( id BIGINT PRIMARY KEY, ( ), kvp_hstore HSTORE ); June 7, 2011 University of Applied Science Rapperswil 10

11 Performance Benchmark Design Test Data 12 different data sets with the following amount of records Each data set testes twice (once with index and once without) GiST (Generalized Search Tree) is used for hstore (basis for B-Tree and R-Tree) Hence 12 [# of length] 2 [# of types] 2 [# of indices] 3 [warm start] =144 [cicles] June 7, 2011 University of Applied Science Rapperswil 11

12 Test Data Test Data Schema Column id : integer, sequence surname : Text forename : Text Description Mandatory. A unique sequence identifier. Mandatory. A fancy name. Optional: A fancy name. Can be empty to have a variable KVP length. zip : Integer Optional: A number between 1000 and comment : Text Optional: A dummy text. Example 1,cucyp,ecnalehad,6593,lorem ipsum dolor sit amet 2,kasarzyc,,6593, June 7, 2011 University of Applied Science Rapperswil 12

13 Test Environment June 7, 2011 University of Applied Science Rapperswil 13

14 Test Environment Technical Specification Intel(R) Xeon(R) CPU 2.27GHz 64-bit 3CPUs, 4 cores and 8 threads 24 GB RAM Software Ubuntu LTS PostgreSQL Python 2.6.5, Numpy, Scipy, Matplotlib No tuning of Software June 7, 2011 University of Applied Science Rapperswil 14

15 BENCHMARK MAY 2011 Results Findings Analyze statements Functionality of hstore Conclusion June 7, 2011 University of Applied Science Rapperswil 15

16 Results June 7, 2011 University of Applied Science Rapperswil 16

17 Results June 7, 2011 University of Applied Science Rapperswil 17

18 Results June 7, 2011 University of Applied Science Rapperswil 18

19 Results June 7, 2011 University of Applied Science Rapperswil 19

20 Results June 7, 2011 University of Applied Science Rapperswil 20

21 Results June 7, 2011 University of Applied Science Rapperswil 21

Findings Table size hstore: array entries tuples KVP tuples value in the, whereas array value null Explain Analyze for KVP 1. Seq Scan on bench_kvp (cost=229.38..437.

22 Findings Table size hstore: array entries tuples KVP tuples value in the, whereas array value null Explain Analyze for KVP 1. Seq Scan on bench_kvp (cost= rows=3 width=60) (actual time= rows=2 loops=1) 2. Filter: (bench_id = $0) 3. InitPlan 1 (returns $0) 4. -> Seq Scan on bench_kvp (cost= rows=1 width=8) (actual time= rows=1 loops=1) 5. Filter: ((key = 'id'::text) AND (value = '1735'::text)) 6. Total runtime: ms June 7, 2011 University of Applied Science Rapperswil 22

Findings Explain Analyze for KVP with index on attribute key 1. Seq Scan on bench_kvp (cost=199.48..406.88 rows=3 width=60) (actual time=2.268..2.730 rows=2 loops=1) 2. Filter: (bench_id = $0) 3.

23 Findings Explain Analyze for KVP with index on attribute key 1. Seq Scan on bench_kvp (cost= rows=3 width=60) (actual time= rows=2 loops=1) 2. Filter: (bench_id = $0) 3. InitPlan 1 (returns $0) 4. -> Bitmap Heap Scan on bench_kvp (cost= rows=1 width=8) (actual time= rows=1 loops=1) 5. Recheck Cond: (key = 'id'::text) 6. Filter: (value = '1735'::text) 7. -> Bitmap Index Scan on kvpidx (cost= rows=2499 width=0) (actual time= rows=2500 loops=1) 8. Index Cond: (key = 'id'::text) 9. Total runtime: ms June 7, 2011 University of Applied Science Rapperswil 23

Findings Explain Analyze for KVP with combined index 1. Seq Scan on bench_kvp (cost=8.27..215.91 rows=3 width=60) (actual time=1.376..1.954 rows=5 loops=1) 2. Filter: (bench_id = $0) 3.

24 Findings Explain Analyze for KVP with combined index 1. Seq Scan on bench_kvp (cost= rows=3 width=60) (actual time= rows=5 loops=1) 2. Filter: (bench_id = $0) 3. InitPlan 1 (returns $0) 4. -> Index Scan using kvpidx2 on bench_kvp (cost= rows=1 width=8) (actual time= rows=1 loops=1) 5. Index Cond: ((key = 'id'::text) AND (value = '1735'::text)) 6. Total runtime: ms 7. (6 rows) June 7, 2011 University of Applied Science Rapperswil 24

Findings Explain Analyze for hstore 1. Seq Scan on bench_hstore (cost=0.00..213.72 rows=45 width=40) (actual time=1.318..1.778 rows=1 loops=1) 2.

33 rows=2 width=218) (actual time=0.481..0.534 rows=1 loops=1) 2. Recheck Cond: (bench_hstore @> '"id"=>"1735"'::hstore) 3. -> Bitmap Index Scan on hidx_2_5k (cost=0.00..4.27 rows=2 width=0) (actual time=0.

25 Findings Explain Analyze for hstore 1. Seq Scan on bench_hstore (cost= rows=45 width=40) (actual time= rows=1 loops=1) 2. Filter: ((bench_hstore -> 'id'::text) = '1735'::text) 3. Total runtime: ms Explain Analyze for hstore with index 1. Bitmap Heap Scan on bench_hstore (cost= rows=2 width=218) (actual time= rows=1 loops=1) 2. Recheck Cond: '"id"=>"1735"'::hstore) 3. -> Bitmap Index Scan on hidx_2_5k (cost= rows=2 width=0) (actual time= rows=70 loops=1) 4. Index Cond: '"id"=>"1735"'::hstore) 5. Total runtime: ms June 7, 2011 University of Applied Science Rapperswil 25

Functionality of hstore Buffers the whole hstore Each hstore key value pair knows: its position in the string its length value and its length Hstore data type 1.

26 Functionality of hstore Buffers the whole hstore Each hstore key value pair knows: its position in the string its length value and its length Hstore data type 1. CREATE TYPE hstore ( INTERNALLENGTH = -1, INPUT = hstore_in, OUTPUT = hstore_out, RECEIVE = hstore_recv, SEND = hstore_send, STORAGE = extended ); June 7, 2011 University of Applied Science Rapperswil 26

linked to a PostgreSQL function 1. CREATE OR REPLACE FUNCTION fetchval(hstore,text) 2.

27 Functionality of hstore -> as exemplary operator 1. CREATE OPERATOR -> ( LEFTARG = hstore, RIGHTARG = text, PROCEDURE = fetchval ); Procedure is linked to a PostgreSQL function 1. CREATE OR REPLACE FUNCTION fetchval(hstore,text) 2. RETURNS text 3. AS 'MODULE_PATHNAME','hstore_fetchval 4. LANGUAGE C STRICT IMMUTABLE; June 7, 2011 University of Applied Science Rapperswil 27

Functionality of hstore PostgreSQL function is linked to a C method hstore_fetchval method returns the value by calling get_val method, which loops over the buffer and returns the position Example id

28 Functionality of hstore PostgreSQL function is linked to a C method hstore_fetchval method returns the value by calling get_val method, which loops over the buffer and returns the position Example id : BIGINT kvp_hstore : HSTORE 1 zip => 8000, surname => ebsaveq 2 zip => 6489, surname => epofod 3 zip => 8000, surname => kjuefs 1. SELECT hstore(bench_hstore)-> surname FROM bench_hstore WHERE hstore(bench_hstore)-> zip = 8000 ; June 7, 2011 University of Applied Science Rapperswil 28

29 Conclusion For small data sets (< 500 records) KVP is preferable However 500 records is easily exceeded Changing schema involves huge effort Transposing data Changing database table schema Possibly refactoring software to new schema If unsure about size use hstore as data type KVP is only 0.45 ms faster at 500 records June 7, 2011 University of Applied Science Rapperswil 29

30 Hindi Russian Traditional Chinese Thank You Thai Gracias Spanish Grazie Italian Arabic English Simplified Chinese Obrigado Brazilian Portuguese Danke German Merci French Tamil Japanese Korean June 7, 2011 University of Applied Science Rapperswil 30

Key/Value Pair versus hstore - Benchmarking Entity-Attribute-Value Structures in PostgreSQL.

Key/Value Pair versus hstore - Benchmarking Entity-Attribute-Value Structures in PostgreSQL. Michel Ott June 17, 2011 University of Applied Science Rapperswil 1 What is KVP? A key-value pair (KVP) is a