PostgreSQL/Jsonb. A First Look

Transcription

1 PostgreSQL/Jsonb A First Look

2 About Me Started programming in 1981 Owner of Enoki Solutions Inc. Consulting and Software Development Running VanDev since Oct 2010

3 Why PostgreSQL? Open Source Feature Rich Mature So much better than MySql MongoDB has issues Only atomic at the document level

4 Why Json? Blame Javascript But, in a DB context Data locality Data atomicity without transaction overhead? Fancy blob?

5 Jsonb? A binary format PostgreSQL specific In theory faster to modify Generally smaller to store Indexable!

6 1st Observation Always use Jsonb if you re going to have the db do anything with it Json can t be indexed

7 Make everything jsonb? CREATE TABLE tst ( id UUID NOT NULL, data JSONB DEFAULT '{}'::jsonb NOT NULL );

8 A quick aside on UUIDs Structure your UUIDs (128 bits) as follows Time (ms since epoch, 44 bits, >557 years) If generating more than 2 12 /ms allow this to drift forward If it ends up being a problem, it won t be your problem Sequence (12 bits = 4096/ms) Node (12 bits = 4096 nodes) Expect up to ~1s of time drift when using ntpd Random (60 bits, 1% collision/ million) Set per ms Why? Ids generated at the same time share locality Faster inserts

9 Make everything jsonb? CREATE TABLE tst ( id UUID NOT NULL, data JSONB DEFAULT '{}'::jsonb NOT NULL ); What happens when you modify data? The whole field is updated If data is large that can be very slow

10 2nd Observation Consider partitioning into sections CREATE TABLE tst ( id UUID NOT NULL, section_name VARCHAR(128), data JSONB DEFAULT '{}'::jsonb NOT NULL ); Updates to data are smaller now Updates by id are no longer atomic across sections unless you use transactions!

11 Indexing CREATE UNIQUE INDEX idx_tst_id ON tst USING btree (id); CREATE UNIQUE INDEX idx_tst_id_section_name ON tst USING btree (id, section_name); CREATE INDEX idx_tst_id_section_name_data ON tst USING btree (id, section_name, data); CREATE INDEX idx_tst_section_name_data_tags ON tst USING btree (section_name, ((data->>'tags') :: TEXT)); CREATE INDEX idx_tst_section_name_data_count ON tst USING btree (section_name, ((data->>'count') :: INT8)); Looks funny doesn t it.

12 Some test data WITH A AS ( INSERT INTO "tst" VALUES (' ', 'meta', '{"tags":["a","b","c"], "count":10}'),(' ', 'meta','{"tags":["a","d","c"], "count":1}') ON CONFLICT DO NOTHING RETURNING * ) SELECT * FROM A; BTW, WITH is awesome

13 Did it work? EXPLAIN SELECT * FROM tst WHERE section_name='meta' ORDER BY (data->'count'); Sort (cost= rows=1 width=354) Sort Key: ((data -> 'count'::text)) -> Index Scan using idx_tst_section_name_data_count on tst (cost= rows=1 width=354) Index Cond: ((section_name)::text = 'meta'::text) EXPLAIN SELECT * FROM tst WHERE section_name='meta' ORDER BY ((data->>'count')::int8); Index Scan using idx_tst_section_name_data_count on tst (cost= rows=1 width=330) Index Cond: ((section_name)::text = 'meta'::text) SELECT * FROM tst WHERE section_name='meta' ORDER BY ((data->>'count')::int8) meta {"tags": ["a", "d", "c"], "count": 1} meta {"tags": ["a", "b", "c"], "count": 10}

14 Updating Count WITH X AS ( UPDATE tst SET data = jsonb_set(data, '{count}', to_jsonb(((data ->> 'count') :: INT8) + 1 :: INT8), FALSE) WHERE section_name='meta' and data? 'count' AND data -> 'tags'? 'd' RETURNING * ) SELECT * FROM X; meta {"tags": ["a", "d", "c"], "count": 2}

15 What about tags? EXPLAIN SELECT * FROM "tst" WHERE section_name='meta' and "data" -> 'tags'? 'b'; Index Scan using idx_tst_section_name_data_count on tst (cost= rows=1 width=32) Index Cond: ((section_name)::text = 'meta'::text) Filter: ((data -> 'tags'::text)? 'b'::text) SELECT * FROM "tst" WHERE section_name='meta' and "data" -> 'tags'? 'b'; meta {"tags": ["a", "b", "c"], "count": 10} Search within an array is linear?

16 Gin anyone? DROP TABLE tst; CREATE TABLE tst ( data JSONB DEFAULT '{}'::jsonb NOT NULL ); CREATE INDEX idx_tst_data ON tst USING GIN ((data->'tags')); EXPLAIN SELECT * FROM "tst" WHERE "data" -> 'tags'? 'b'; Bitmap Heap Scan on tst (cost= rows=1 width=32) Recheck Cond: ((data -> 'tags'::text)? 'b'::text) -> Bitmap Index Scan on idx_tst_data (cost= rows=1 width=0) Index Cond: ((data -> 'tags'::text)? 'b'::text)

17 Add back section_name CREATE TABLE tst ( section_name VARCHAR(128), data JSONB DEFAULT '{}'::jsonb NOT NULL ); CREATE INDEX idx_tst_data ON tst USING GIN (section_name, (data->'tags')); sql> CREATE INDEX idx_tst_data ON tst USING GIN (section_name, (data->'tags')) [ :19:21] [42704] ERROR: data type character varying has no default operator class for access method "gin" Hint: You must specify an operator class for the index or define a default operator class for the data type. D oh

18 btree_gin? CREATE EXTENSION btree_gin; CREATE TABLE tst ( section_name VARCHAR(128), data JSONB DEFAULT '{}'::jsonb NOT NULL ); CREATE INDEX idx_tst_data_1 ON tst USING gin (section_name, (data->'tags')); EXPLAIN SELECT * FROM "tst" WHERE section_name = 'meta' and "data" -> 'tags'? 'b'; Seq Scan on tst (cost= rows=1 width=306) Filter: (((section_name)::text = 'meta'::text) AND ((data -> 'tags'::text)? 'b'::text)) Worse?!

19 No, we need more data create or replace FUNCTION tmpf() RETURNS void AS $$ declare i INTEGER; BEGIN i = 0; while i< loop i = i + 1; insert into tst values ('meta','{"tags":["a"]}'); insert into tst values ('meta','{"tags":["b"]}'); insert into tst values ('meta','{"tags":["b","c"]}'); insert into tst values ('meta','{"tags":["c"]}'); end loop; END $$ LANGUAGE plpgsql;

20 Before and After index EXPLAIN ANALYSE SELECT * FROM "tst" WHERE section_name = 'meta' and "data" -> 'tags'? 'a'; Seq Scan on tst (cost= rows=1 width=306) (actual time= rows= loops=1) Filter: (((section_name)::text = 'meta'::text) AND ((data -> 'tags'::text)? 'a'::text)) Rows Removed by Filter: Planning time: ms Execution time: ms CREATE INDEX idx_tst_section_name_data ON tst USING gin (section_name, (data->'tags')); EXPLAIN ANALYSE SELECT * FROM "tst" WHERE section_name = 'meta' and "data" -> 'tags'? 'a'; Bitmap Heap Scan on tst (cost= rows=400 width=32) (actual time= rows= loops=1) Recheck Cond: (((section_name)::text = 'meta'::text) AND ((data -> 'tags'::text)? 'a'::text)) Heap Blocks: exact=3054 -> Bitmap Index Scan on idx_tst_section_name_data (cost= rows=400 width=0) (actual time= rows= loops=1) Index Cond: (((section_name)::text = 'meta'::text) AND ((data -> 'tags'::text)? 'a'::text)) Planning time: ms Execution time: ms

21 Any better without section? CREATE INDEX idx_tst_data_tags ON tst USING GIN ((data -> 'tags')); EXPLAIN ANALYSE SELECT * FROM "tst" WHERE "data" -> 'tags'? 'a'; Bitmap Heap Scan on tst (cost= rows=400 width=32) (actual time= rows= loops=1) Recheck Cond: ((data -> 'tags'::text)? 'a'::text) Heap Blocks: exact=3054 -> Bitmap Index Scan on idx_tst_data_tags (cost= rows=400 width=0) (actual time= rows= loops=1) Index Cond: ((data -> 'tags'::text)? 'a'::text) Planning time: ms Execution time: ms

22 Go big Add 4 million rows 1 row with tag e EXPLAIN ANALYSE SELECT * FROM "tst" WHERE section_name = 'meta' and "data" -> 'tags'? 'e'; Planning time: ms Execution time: ms CREATE INDEX idx_tst_section_name_data ON tst USING gin (section_name, (data->'tags')); EXPLAIN ANALYSE SELECT * FROM "tst" WHERE section_name = 'meta' and "data" -> 'tags'? 'e'; Planning time: ms Execution time: ms Looks like it works.

23 3rd Observation Writes (updates) get slow GIN index updates are that not fast 4 million inserts too ~3 minutes on my machine

24 Summary It s weird, but it works Need to specify type a lot Need to learn about indexes Need to watch out for document size Need to watch out for index update time WITH is awesome

25 Q&A